Full Disk Access is required to access Time Machine backups in Mojave

I’ve been struggling since Mojave came out to deal with it’s over-bearing expansion of SIP (“System Integrity Protection”), which is basically a super-root notion that blocks access – even to root – to lots of basic parts of the system, including obvious & mostly sensible ones like /System and /Library, but also less usefully things like any & all Time Machine backups.

Blocking access to Time Machine makes it very difficult to actually use Time Machine, since it’s then difficult to retrieve files from a backup (you have to then use the stupid ‘warp’ Time Machine interface, which is slow, ugly, and buggy).

Luckily, it turns out there is a fairly simple solution that isn’t disabling SIP entirely (which requires multiple reboots in order to do, so is typically quite disruptive & slow). It appears that any application granted Full Disk Access (System Preferences → Security & Privacy → Full Disk Access) can read Time Machine backups.

In case you’re unfamiliar, the symptoms of this problem include:

  • Being unable to navigate into Time Machine backups in the Open / Save / etc dialogs.
  • Being unable to see – through ls or similar tools – the contents of Time Machine backups via Terminal.
  • Apps reporting errors like “The file “Foo” couldn’t be opened because you don’t have permission to view it” or bluntly “Operation not permitted” when trying to read something in a Time Machine backup.

There’s a strange & ironically very bad security quirk though – curiously, any tools run via Terminal inherit Terminal’s access (or lack thereof) to Full Disk Access. They don’t use whatever setting might be specified for them in the Security & Privacy preferences. This is pretty baffling, as it means to give Full Disk Access to anything you run via Terminal, you have to give it to everything you run via Terminal. Anything you specifically give Full Disk Access won’t actually receive it if it happens to be launched via the Terminal (which confused me for a while, since it’s so unintuitive).

I’m guessing whatever mechanism enforces all this so-called security is based in LaunchServices or somesuch – while the Finder and most things in general will launch apps via LaunchServices, as detached & independent process sessions, Terminal doesn’t – everything it runs, from the shells down, run under it in the process hierarchy, and seemingly share its security & privacy settings.

ffmpeg can produce pseudo-corrupt audio when ‘copy’ing to an MP4 container

I’ve been using ffmpeg to trim clips from a trail camera, as most of the time there’s only a few seconds of anything interesting in frame out of the 30+ seconds of video it records each time, but I don’t want to re-encode them and lose video quality as a result (or balloon file sizes tremendously with a lossless video coding).  Keeping the whole 30 seconds is not just unnecessary and makes viewing the videos much more tedious, but wasteful of storage space as the encoding quality from the trail camera is very inefficient (file sizes are many times larger than they should be for the quality – clearly the H.264 encoder used in the trail camera is very cheap and very bad at its job).

I was originally doing something like:

ffmpeg -ss 00:07 -t 00:03 -i "IMG_0164.MP4" -async 1 -c copy "IMG_0164_TRIMMED.MP4"

The resulting trimmed MP4s play just fine in Quicktime, the Finder – anywhere that uses Apple’s decoding libraries (though I didn’t test iOS).

However, in VLC, or Lightroom, the audio is completely corrupt – just incoherent noise.  In Lightroom the video doesn’t even play correctly, because of Lightroom’s stupid habit of re-encoding the video & audio into internal caches – apparently their video decoder is somehow thrown off by the audio channel issues, too.

After much trial and error and many dead-ends (thank you completely bogus & wrong Stack Overflow threads… sigh) I eventually realised that the problem is apparently simply that Lightroom, VLC, etc get offended when you include pcm_s16le audio in an MP4.  ffmpeg itself says that’s not a valid audio codec for the MP4 container, iff you explicitly tell it to use that as the codec.  If you’re just copying from an existing audio / video file, however, it makes no mention at all of the concern.  Sigh.

So the apparent solution is simply to switch to the MOV container format instead.

ffmpeg -ss 00:07 -t 00:03 -i "IMG_0164.MP4" -async 1 -c copy "IMG_0164_TRIMMED.MOV"

The encoded bits remain identical, but the MOV container apparently accepts PCM audio where MP4 does not.  VLC, Lightroom, etc are now happy (and Quicktime et al remain happy).

(another possibility is that the ‘incompatibility’ is related to MP4 levels or some other such junk… I didn’t try deciphering or exploring that)

It’s frustrating that VLC & Lightroom can’t handle this when clearly it’s technically possible (witness Quicktime), and worse they don’t even properly recognise that they’re not handling it properly – they just play completely corrupt audio that’s literally painful on the ears.

It’s also very curious that the trail camera uses PCM audio if that’s not valid in an MP4 container.  It’s downright bizarre that VLC & Lightroom can play the unmodified MP4s straight from the trail camera, even though they use the same purportedly invalid audio codec… somehow something ffmpeg is doing during its transmutation is making them angry.  I was unable to determine what that might be, though, through trial-and-error with ffmpeg command line options & rudimentary examination of the input & output files.

P.S.  An alternative is to bitwise-copy only the video stream (i.e. change -c copy to -c:v copy), and let VLC transcode the audio into its default AAC for the MP4 container.  That probably wouldn’t be a problem for me in my case – the audio from trail cameras is pretty crappy to begin with – but at the same time the audio tracks in these files are insignificant in size, so re-encoding them (and lossy as AAC) is pointless to saving disk space.