Patch Notes #123 — This Is Not a Drill (It Was a Drill)

Last Saturday morning, every phone in Hawaii received: “BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.” For thirty-eight minutes, it stood. People put children in storm drains. Said goodbye. And the cause, when it emerged, was the most professionally humbling artifact of the year: an employee running a shift-change drill selected the wrong item from a dropdown menu. The real-alert option lived adjacent to the test option, same list, similar labels, with a confirmation dialog that (like all confirmation dialogs) had been trained into muscle-memory clickthrough. No cancel template existed; one had to be composed, mid-crisis, over 38 minutes.

This blog has a whole shelf for this: Joyent’s datacenter reboot, the S3 typo. The operator is never the bug; the dropdown is the bug. Test and production sharing a menu is the bug. The missing undo is the catastrophic bug. Every engineer I know read that story and went hunting through their own admin panels for dropdowns where “drill” sits one arrow-key from “apocalypse.” We found two. Fixed by Wednesday.

Elsewhere: crypto’s biblical correction has begun in earnest. Bitcoin roughly halved from December’s peak, altcoins worse, my barber gone quiet. The date, previously unknowable, is looking a lot like “the top was the futures launch.”

TIL: forcing functions in UI, physical interlocks, typed confirmations (“type the environment name to proceed”), and separated blast radii. Guardrails over discipline, now with a missile-alert case study for the training deck.