Your system has run out of application memory HUR HUR HUR

I hate this dialog with the fire of a thousand suns.

When this appears, it basically means one (or both) of two things:

  1. Some application went nuts and chewed through all your memory and/or disk space.
  2. macOS got itself into a darkly comical & embarrassing deadlock.

Quitting any of the listed applications is rarely the correct move.  It’s often enough the case that none of them are the root cause, and you can kill all of them if you want, but it won’t fix the problem.

One important thing to clarify first, though, is that this dialog does not necessarily use the term ‘memory’ in the conventional sense – i.e. RAM.  It can also refer to disk space.  Unfortunately it doesn’t bother to distinguish between the two, which is particularly stupid of it since any possible resolution of the issue is highly dependent on which of the two cases it in fact is.

Thank goodness for iStatMenus, though, which in the most recent incident showed that I had ~20 GiB of RAM completely free (not even inactive, actually outright free).  So immediately that rules out what the daft bloody dialog’s actually saying.

The worst thing about all this is when it’s #2 the occurs.  For example, I had Lightroom do a 63-image panorama merge.  As Lightroom is a gross memory pig when doing panorama merging, it consumed something like 40 GiB of memory.  Which caused a bunch of stuff to page to disk.  Which consumed all the disk space.  Which led to that obnoxious dialog.  Which further led to macOS in its infinite fucking wisdom ‘pausing’ (SIGSTOPing) almost all running programs, including evidently whatever daemon actually handles paging.  Thus when Lightroom actually completed the panorama merge and released all that memory, I now had 20 GiB of free memory and the system refused to use any of it to page back in all that memory it’d paged out.  Because it was out of disk space.

The only solution – short of hard rebooting and hoping it resolves itself – was to delete a bunch of files I actually do still want, but which will now have to be recovered from a backup.  Great job macOS, thanks for all your help.

Of course, even once you do that and recover the system from the derpeche mode it put itself into, it won’t actually unpause any of the shit it broke.  You have to do that manually.  It pretends you can do that via that dialog that started the whole thing – assuming you left it open the entire time, blocking your view as you actually help the situation – but that only shows user-visible applications, not all the other system & background processes that it also rudely halted.

So, simple tip for resuming everything:

sudo killall -CONT -m '.'

Elegant, after a fashion.  Though every time, it reminds me that whomever named it ‘killall’ was either not very friendly or not very wise.

Note that the system will probably still be a bit broken in places, as despite what macOS thinks, you can’t just blindly pause random system tasks and not have things get really, really confused.  A reboot is always wise after seeing this dialog, to properly undo its fuckery.

Nikon SnapBridge

Finally.

Nikon have released the SnapBridge app so that the much-touted Bluetooth+Wifi capability of the D500 can actually be used.  A mere eight months after it was announced.  Fuck you too Nikon.

However, as I’d clearly forgotten, it’s not very useful anyway.  It doesn’t work with raws, you see.  Doesn’t even acknowledge that they’re in the camera, on the card.  It took me twenty minutes of screwing around with the app, wondering why it was so completely broken and dysfunctional, before I stumbled upon a tech support article for it buried half a dozen layers deep inside Nikon’s website (yes, there’s essentially no documentation within the app itself).

It does appear to at least work for geotagging & time sync, which is something.  Something Nikon could have put in natively for a $1 GPS receiver, and then not have to kill my iPhone battery to accomplish rudimentary tasks.

The almost saving grace of the D500 is its speed – specifically the UHS-II support, which helps it clear out its buffer extra snappy, given a decent SD card.  That means I can turn on NEF+JPEG without much concern about slowing down burst shooting, and only marginal concern about the wasted SD card space.

But it’s only almost saved by it.

The problem, you see, is that even if you abuse the NEF+JPEG option to yield little JPEG turds on your SD card – and even though those JPEGs can be surprisingly decent quality, even on ‘Small’ and ‘Basic’ settings – in NEF+JPEG mode the camera insists on using the JPEG version for all in-camera playback.  It becomes completely impossible to view the actual NEF.

Now, granted when ‘viewing’ NEFs in-camera you’re only getting the JPEG preview that’s built into them anyway, but still – it’s at least a decent quality, full-size preview.  You can at least zoom all the way in.  Not so if your JPEG turds are not full-size.

Which might be a good enough option, if one is willing to waste up to 50% of your space saving full-size JPEGs alongside the NEFs.

But, SnapBridge transfers the images via Bluetooth only.  Even when you’ve configured it to bring over the originals, at up to 10 MB each.  It can take minutes to transfer a single image of that size at Bluetooth speeds – I know, I accidentally proved it empirically.

Now, you can limit the transfer to 2 MP versions of those JPEGs, but 2 MP is tiny, even by Shitagram standards.  The ‘Small’ JPEGs the D500 saves natively are 5.2 MP, for point of reference.

So the 2 MP transfer option – call it “Thumbnails only” – is not a practical or useful option.

So we’re back to having to use full-size JPEGs, alongside the real photos (the NEFs).

And remember the prior point about abysmal Bluetooth transfer speeds?  To make SnapBridge’s auto image transfer plausible to use with any frequency – let-alone leave on permanently – you need tiny file sizes.  Even on the highest compression setting (vanilla ‘Basic’) the 21 MP JPEGs are several megabytes.  Only by using the ‘Small’ image size – which is frankly still good enough for Instagram types – can you get the sizes into the sub-MB range, and transfer times down to ‘merely’ a few seconds per photo.

So you’re stuck between a rock and a hard place.  The net result is that the whole image download thing’s kinda horrible and useless to me.  Which makes me sad, because it could easily have been implemented much better.

The CamRanger remains a significantly better experience in almost every respect – the main detractor being the additional monetary cost it imposes.

Silent data corruption

Alternate title:  Apple’s file system engineers are sadly naive.

I was quite disappointed to see that APFS isn’t even trying to provide data integrity.  Data integrity is kind of step 0 of any file system, and checksums or use of ECC is pretty much standard in modern & leading-edge file systems.  APFS doesn’t want to be one of those, it seems.

Case in point why this matters:

I have a bunch of old backup drives, because drives are cheap and until recently I could just buy a new one once the current one filled, instead of ever deleting a backup.  Periodically I go back through these old backup drives and do some basic integrity checks (S.M.A.R.T. bad block scans, file system checks, etc).

also run a comparison of key data between those backups and the current versions on my computer, for files which generally shouldn’t change nor disappear – e.g. photos, videos, key documents, etc.

And today I found that at least half a dozen valuable personal videos (and a few photos) were corrupt, in the versions on my computer.  Luckily, the versions in the ancient backups were still good, so I could replace the corrupt ones.

This corruption was completely silent, until my ‘paranoid’ and time-consuming checks discovered it.

It’s far from the first time.  A failing drive years back corrupted a huge portion of my music library – silently, as far as the file system & OS were concerned.  Periodically I’ve discovered photos (of which I have huge numbers – the majority of my data) which have become corrupt at some indeterminate point.  And I’ve of course had file system [metadata] corruption occur many times, sometimes requiring complete erasure of the disk, and recovery or rebuilds from backup (a few times I’ve had to use data recovery software, where backups weren’t available).

Most, if not all, of these issues would have been discovered by even the most trivial file integrity protections, in the file system.

The notion that modern disks somehow magically protect against all silent data corruption is abject poppycock.  They’re more likely to suffer from it than older disks – a byproduct of higher densities and market demand for cheaper, crappier storage products.

And the implicit assertion that Apple’s file system driver, and kernel overall, are somehow completely free of bugs… is just batshit crazy.

Addendum

Since Apple aren’t interested in protecting anyone’s valuable personal data, I’m on the look-out for other options.  Manual use of shasum is one, for now, but a more streamlined and fool-proof system would be better.  Alas, none seems to exist1.  Yet.

  1. There is chkbit, but it relies on MD5… probably acceptable for this use case, but needless in the face of decades of better hash algorithms.  And it’s written in JavaScript.  Ew.