The great S3 outage of 2017: on February 28th, an AWS engineer debugging the billing system typo’d a playbook command and removed way more capacity than intended from us-east-1’s S3, and it turned out half the internet, including AWS’s own status dashboard, lives in that bucket-shaped basket. Four hours of a broken web. Amazon’s postmortem is admirably specific: the tool allowed too-big removals (now capped), and a subsystem hadn’t been restarted in years and took ages to cold-boot. Zero blame placed on the human, all of it on the system that let one keystroke go nuclear, the Joyent lesson from 2014 at 100x scale. If a typo can take down the internet, the typo isn’t the bug. Also: host your status page somewhere else. Everyone. Please. ...

The most consequential engineering document of the fortnight wasn’t a paper or a postmortem, it was a personal blog post. Susan Fowler, an engineer, published “Reflecting on One Very, Very Strange Year at Uber”: calm, precise, receipts-grade documentation of harassment reported and buried, HR processes that protected the wrong people, and a culture rotting from its incentives. It detonated. Uber launched an external investigation within days; the industry’s whisper networks went loud; and every eng leadership team I know (including ours, forty people, no excuses) is asking “could that happen here and would we know?” The technical-adjacent lesson: she wrote it like an incident report, with a timeline, evidence, escalation paths, and systemic analysis over villain-hunting, and that is why it couldn’t be dismissed. Documentation is power. It always was. ...

Entry one hundred. Four years and a month of fortnightly patch notes, and the office spent the milestone week debating the greatest comebacks in sports history. We kept returning to Manchester United’s 1999 Champions League final. Down 1-0 in the 90th minute (win probability models would have shown Bayern Munich above 99%), and then the most dramatic three minutes in football history, sealed before the whistle. Teddy Sheringham sweeping in the equalizer, Ole Gunnar Solskjær stabbing the winner into the roof of the net like a man defying the concept of physics out of spite. ...

Switch details landed: $299, March 3rd, and the new Zelda, Breath of the Wild, ships day one. Preorder: executed within ninety seconds of the stream ending, via the import route, naturally — Nintendo’s official map still doesn’t include India, but the cousin network has never once missed a launch. The lineup beyond Zelda is thin and the paid-online announcement stung, but Nintendo is doing the thing great engineering orgs do after a failure: the Wii U postmortem is visible in every Switch decision. Confusing message? Now one clear concept (the console you take with you). Starved third-party support? Cartridges, portable dev kits, Unreal Engine support. Weak launch title? The most anticipated Zelda ever. You can read the incident review in the product. That’s how you know it was a real review and not theater. ...

Year five begins. CES verdict from every floor report: Alexa won, and Amazon barely even had a booth. Alexa in fridges, cars, lamps, showerheads, hundreds of third-party devices, all voice-piping to Amazon’s cloud. While Google and Apple were building assistants as features of their own hardware, Amazon quietly built one as a platform and let everyone else do the manufacturing. The Echo looked like a gimmick in 2014; it’s an ecosystem in 2017. The platform play is invisible until it’s inevitable. ...

Ninety-seven entries. Four years, zero missed. The streak is my longest-running production system. 2016’s thesis, and it wasn’t subtle: every model broke. Leicester City won the Premier League at 5000-to-1, the single least probable result in team sports history, and I never even blogged it because I didn’t believe it in May. Hamilton lost the title despite a rain-delayed heroic run as Rosberg took the crown and retired five days later. England fell to Iceland’s defensive block. Brexit beat the polls; the election beat the polls; AlphaGo beat the species. If your priors survived 2016 intact, you weren’t paying attention or you weren’t betting. The out-of-distribution events are the only ones that matter, and this year was all distribution tails, all the time. ...

Amazon announced Go this week: a real grocery store, opening in Seattle, with no checkout. Walk in, grab things, walk out; cameras and sensors and ML figure out what you took and bill your account. “Just Walk Out technology,” which is either the best product name or the best shoplifting instruction ever printed. The demo video looks like sci-fi; the employee-beta caveats suggest the edge cases (occlusion, shared carts, indecisive shelf-returners) are where the bodies are buried, because they always are, and the demo is the dream. But note the trajectory: Amazon spent a decade making buying invisible online (one-click, Dash buttons, Alexa), and now it’s compiling that philosophy into physical space. Friction is their sworn enemy in every dimension of reality. ...

The post-election story consuming tech: fake news as a business model. Teenagers in Macedonia ran profitable content farms of invented headlines because the feed pays per engagement and outrage is the highest-yield emotion. Zuck’s initial “crazy idea” dismissal aged badly within days; Facebook and Google both scrambled to cut ad-network access for fake-news sites this fortnight. Here’s the uncomfortable engineering truth the think-pieces circle without landing: nobody wrote if (fake) promote(). They wrote “maximize engagement,” and the optimizer found the exploit, because optimizers always find the exploit, that’s what they’re for. Tay learned from trolls in sixteen hours; the feed learned from human nature over a decade. Same bug. Objective functions are wishes made to a very literal genie. ...

The US election happened Tuesday, and the polling models faceplanted again — the out-of-distribution lesson, now officially a genre. The industry’s postmortems this time aren’t about servers; they’re about feeds: filter bubbles, engagement optimization, fake-news economics. Those are our systems. The reckoning about what recommendation engines optimize for has left the conference track and entered the congressional calendar. More on that as it develops, which it will, for years. F1’s title fight, meanwhile, is compressing beautifully: Hamilton won in Mexico a week and a half ago to keep the math alive, but Rosberg only needs to cruise. Two races left — São Paulo this Sunday, then the Abu Dhabi decider. The group chat has pre-committed to no eulogies until the math is dead. (I have drafted one anyway. Blameless.) ...

Friday, someone turned off the internet’s phone book. A massive DDoS hit Dyn, the managed-DNS provider behind Twitter, Netflix, Reddit, Spotify, and GitHub, and for hours, half the household-name web wouldn’t resolve for much of the US. The weapon: Mirai, a botnet of hacked IoT devices. Security cameras. DVRs. Baby monitors. Default passwords, internet-exposed, conscripted by the hundreds of thousands. The IoT prophecy is ahead of schedule: the S in IoT stands for security, and the things we connected without thinking are now infrastructure-grade artillery pointed at whatever choke point volunteers. ...