The great S3 outage of 2017: on February 28th, an AWS engineer debugging the billing system typo’d a playbook command and removed way more capacity than intended from us-east-1’s S3, and it turned out half the internet, including AWS’s own status dashboard, lives in that bucket-shaped basket. Four hours of a broken web. Amazon’s postmortem is admirably specific: the tool allowed too-big removals (now capped), and a subsystem hadn’t been restarted in years and took ages to cold-boot. Zero blame placed on the human, all of it on the system that let one keystroke go nuclear, the Joyent lesson from 2014 at 100x scale. If a typo can take down the internet, the typo isn’t the bug. Also: host your status page somewhere else. Everyone. Please.
And then March 3rd delivered the good kind of history: the Switch launched with Breath of the Wild, and it is, I say this with a senior engineer’s calibrated restraint, the best game I have ever played. Nintendo threw out every Zelda convention (the postmortem-driven redesign extends into the game design) and built a world that rewards pure curiosity. I climbed a mountain because it was there. There was a shrine. There’s always a shrine. Sales are already historic; the “doomed” branch is deleted.
TIL: cold-start debt. Systems that haven’t restarted in years are systems you no longer know how to start. Reboot your things on purpose, in daylight, quarterly. We scheduled ours for Tuesday.