Truly deleting ‘removed’ files from Lightroom

When you tell Lightroom to deleted rejected photos, it pops up a dangerous dialog box:

Screen shot of Lightroom dialog asking if you want to actually delete rejected photos, or merely lose track of them

Though it does explain itself well – i.e. if you want to actually delete the photos, you need to click “Delete from Disk” – the default option is that misleading “Remove” button, which doesn’t really remove the files at all – it merely makes Lightroom lose track of them. They’ll still be there on disk, wasting space forever.

And, you can’t directly undo this operation, so if you hit return a little too quickly, or misread the dialog at any point, you’re seemingly pretty screwed (if you have a Lightroom catalog of any significant size).

Luckily, there is a way to find these undead files – that doesn’t require you walking through every single file on disk one by one & comparing against Lightroom’s view of the world.

1In the left-side panel, under the “Folders” section, select all the folders and right-click on them (if you have multiple volumes listed under “Folders”, you’ll have to do this one volume at a time as Lightroom won’t let you select folders across multiple volumes simultaneously). You’ll get a contextual menu:

Screen shot of the contextual menu from right-clicking on an entry in the 'Folders' section of the Lightroom left-side panel

2Click “Synchronize Folder…”. A dialog will appear:

Screenshot of the "Synchronize Folder" dialog

You probably want to uncheck “Remove missing photos from catalog” (if it’s not already disabled) and “Scan for metadata updates”, as those are unrelated to the purpose here and have their own ramifications. Instead, just select “Import new photos” and “Show import dialog before importing”. Then, click “Synchronize”.

3Lightroom’s standard import dialog will now appear, and will slowly sort through all the files in the folder(s) you selected, filtering them down to just those that exist on disk yet are not tracked in Lightroom – e.g. all those rejects you accidentally “Removed” but didn’t really remove previously. You can now review those and see what you’ve got – it’s possible you’ll find in there media you didn’t intend to delete, but rather were somehow misplaced by Lightroom at some point.

You might want to, in the import dialog, change your preview generation setting to ‘Minimal’ in order to minimise import time & wasted preview generation. You could also choose to add some keywords to the imports, e.g. “to be deleted” or “recovered” or “undead”, if you’re not going to just immediately delete them anyway.

In any case, you can now import some or all the undead files. Importing them might seem counter-productive, since the goal here is to delete them – but it’s necessary for the final step…

4Once they’re imported, you can now immediately mark them as rejects and delete all rejects again – this time correctly choosing “Removing from Disk”.

So while it’s a bit roundabout, it does get the job done pretty quickly and easily. Now if only Lightroom would fix that stupid dialog to make the default option the one that actually does what you told Lightroom to do to begin with. 🙄

iCloud ‘Optimize Mac Storage’ breaks the Mojave installer

Yet another example of a really bizarre macOS bug that’s pretty inexcusable as a test escape, given it occurs with the default installation settings on a completely clean OS install.

In short, the Mojave update installer does not work (on High Sierra at least) if you have ‘Optimize Mac Storage’ enabled for iCloud Drive (System Preferences > iCloud pane > iCloud Drive Options… button > Documents tab > Optimize Mac Storage checkbox).

Specifically, the installer reports:

Installation requires downloading important content. That content can’t be downloaded at this time. Try again later.

…and indeed fails to download the actual Mojave update files (the installer app as ‘installed’ via the App Store is merely a 22 MB bootstrapping app, that downloads the actual image only after you run it & start the installation).

Even more obnoxiously, if you use the dosdude1 Mojave Patcher Tool to force-download the entire installer, as soon as it completes the 6.5 GB download and produces the ‘Install macOS Mojave’ app in /Applications, the system deletes the downloaded installation files out from under that app, rendering it just as broken as the official App Store version. Infuriating.

Aside: to be clear, turning off ‘Optimize Mac Storage’ enabled me to produce – and keep – a working installer as downloaded by dosdude1’s tool. I did not verify that it also fixes the regular installer as downloaded via the App Store.

I also ran into the “The recovery server could not be contacted” error message even before all the above, but thankfully that was fixable via the means normally prescribed online – running “sudo ntpdate -u”.

Full Disk Access is required to access Time Machine backups in Mojave

I’ve been struggling since Mojave came out to deal with it’s over-bearing expansion of SIP (“System Integrity Protection”), which is basically a super-root notion that blocks access – even to root – to lots of basic parts of the system, including obvious & mostly sensible ones like /System and /Library, but also less usefully things like any & all Time Machine backups.

Blocking access to Time Machine makes it very difficult to actually use Time Machine, since it’s then difficult to retrieve files from a backup (you have to then use the stupid ‘warp’ Time Machine interface, which is slow, ugly, and buggy).

Luckily, it turns out there is a fairly simple solution that isn’t disabling SIP entirely (which requires multiple reboots in order to do, so is typically quite disruptive & slow). It appears that any application granted Full Disk Access (System Preferences → Security & Privacy → Full Disk Access) can read Time Machine backups.

In case you’re unfamiliar, the symptoms of this problem include:

  • Being unable to navigate into Time Machine backups in the Open / Save / etc dialogs.
  • Being unable to see – through ls or similar tools – the contents of Time Machine backups via Terminal.
  • Apps reporting errors like “The file “Foo” couldn’t be opened because you don’t have permission to view it” or bluntly “Operation not permitted” when trying to read something in a Time Machine backup.

There’s a strange & ironically very bad security quirk though – curiously, any tools run via Terminal inherit Terminal’s access (or lack thereof) to Full Disk Access. They don’t use whatever setting might be specified for them in the Security & Privacy preferences. This is pretty baffling, as it means to give Full Disk Access to anything you run via Terminal, you have to give it to everything you run via Terminal. Anything you specifically give Full Disk Access won’t actually receive it if it happens to be launched via the Terminal (which confused me for a while, since it’s so unintuitive).

I’m guessing whatever mechanism enforces all this so-called security is based in LaunchServices or somesuch – while the Finder and most things in general will launch apps via LaunchServices, as detached & independent process sessions, Terminal doesn’t – everything it runs, from the shells down, run under it in the process hierarchy, and seemingly share its security & privacy settings.