It appears that the standard behaviour for an NSView (NSResponder in general?) is for it to “hold onto” pressed keys. That is, if a key is held down the view will keep getting keyDown: events until the key is lifted, no matter what else happens. So it can lose focus or the app loses frontmost status or anything else – it’ll still sit there spewing keyDown: events.
I don’t know a proper way to change this behaviour. You can work around it by ignoring keyDown: invocations while ![NSApp isActive]. Seems to work, without any obvious downsides. Definitely a hack, though. In my specific case I don’t ever care about repeats, so I could filter them using -[NSEvent isARepeat]. Or ignore repeats iff ![NSApp isActive]. Significantly less hackish, if it fits your requirements.