The great S3 outage of 2017: on February 28th, an AWS engineer debugging the billing system typo’d a playbook command and removed WAY more capacity than intended from us-east-1’s S3 — and it turned out half the internet, including AWS’s OWN STATUS DASHBOARD, lives in that bucket-shaped basket. Four hours of a broken web. Amazon’s postmortem is admirably specific: the tool allowed too-big removals (now capped), and a subsystem hadn’t been restarted in YEARS and took ages to cold-boot. Zero blame placed on the human, all of it on the system that let one keystroke go nuclear — the Joyent lesson (2014, #032-adjacent) at 100x scale. If a typo can take down the internet, the typo isn’t the bug. Also: HOST YOUR STATUS PAGE SOMEWHERE ELSE. Everyone. Please.
And then March 3rd delivered the good kind of history: the SWITCH launched with Breath of the Wild, and it is — I say this with a senior engineer’s calibrated restraint — the best game I have ever played. Nintendo threw out every Zelda convention (the postmortem-driven redesign, #099, extends into the GAME design) and built a world that rewards pure curiosity. I climbed a mountain because it was there. There was a shrine. There’s ALWAYS a shrine. Sales are already historic; the “doomed” branch (#093) is deleted.
TIL: cold-start debt — systems that haven’t restarted in years are systems you no longer know how to start. Reboot your things ON PURPOSE, in daylight, quarterly. We scheduled ours for Tuesday.