Patch Notes #157 — The North, Breathtaking

Liverpool won the Champions League in Madrid twelve days ago and I’m only now able to write about it calmly. The club of Istanbul and years of European heartbreak, the team whose own fanbase ran on Klopp’s belief and heavy-metal football, beat Tottenham 2-0, and an entire city turned out for the parade. Jordan Henderson completes the strangest superstar arc of the era: doubted for years, managed to precisely peak fitness, and delivered the most efficient European run. The opponent didn’t lose so much as run out of redundancy. Kane rushed back from injury into maximum load (the human cost of availability pressure, and the sports world’s ugliest rushing-back decision), and the Tottenham machine failed the way all over-optimized systems fail: not gradually, but all at once, at peak demand, out of spares. Liverpool’s title is deserved and the counterfactual is heavy; both go in the file (the asterisk debate has a mirror twin now). ...

June 13, 2019

Patch Notes #156 — Endings, Blacklists, and the Finals Nobody Predicted

Game of Thrones ended, and the postmortem consensus crystallized fast: the show’s failure was a dependency failure. It outran its source material (the books, the load-bearing upstream, unfinished) and the showrunners chose velocity over coherence for the final seasons, shipping plot destinations without the connective causality that made the early seasons feel inevitable. A petition for a remake hit millions of signatures (absurd; noted; the audience now believes endings are patchable, which is itself a symptom of the industry I work in). The lesson for the file is real though: endings are a distinct engineering discipline, migrations, deprecations, series finales, and organizations staff them with whoever’s left, at minimum enthusiasm, at maximum stakes. The best teams I know assign their best people to endings. The credits determine the memory (peak-end rule; psychology’s TIL for the fortnight). ...

May 29, 2019

Patch Notes #155 — The Decade's Bill Comes to Market

Uber IPO’d Friday, the decade’s defining startup, the one whose name became the “Uber for X” template this blog has tracked since folding-chair Demo Days, and it fell on debut, closing under its offer price, one of the worst first-day performances for a mega-IPO ever. Lyft, public six weeks earlier, has slid ~25% from its debut. The pattern demands honest logging because the whole ecosystem’s incentives run through it: a decade of cheap money (still the base layer of everything) funded growth-over-unit-economics at unprecedented scale, private valuations marked up round after round, and the public market, the first counterparty with no stake in the mythology, just declined to pay the last markup. The private-to-public handoff is a settlement layer, and it settled low. Not a crash; a re-pricing of narrative. SoftBank’s Vision Fund strategy (capital as a moat) gets its first grade, and it’s an incomplete trending toward a C-minus. Watch WeWork, the thesis’s purest expression; its IPO looms this fall, and its S-1 will be the most-read document of the year one way or another. (Prediction filed: it does not go smoothly.) ...

May 14, 2019

Patch Notes #154 — Cathedrals, Backups, and Portals

Notre-Dame burned the evening after last entry posted: the spire falling live on every feed, 850 years of accumulated civilization nearly lost to (per early reports) renovation-work ignition, saved at the last defensible line by firefighters who triaged like incident commanders: the towers hold or everything falls, so the towers get everything. Two files for the archive. One: the relic chain worked. The crown of thorns and the rose windows survived because centuries of custodians maintained evacuation plans for objects, rehearsed across generations; the cathedral had runbooks, older than most nations. Two, the tech-adjacent thread everyone’s sharing: the building was comprehensively laser-scanned years ago (a billion-point cloud, by a scholar who died in 2018), and Ubisoft famously modeled it for Assassin’s Creed Unity, so restoration will lean on digital twins made for scholarship and entertainment. Backups you didn’t know were backups: every faithful copy of a thing is a disaster-recovery asset, whatever it was made for. ...

April 29, 2019

When Automation Fights Back: Split Brains, Lightning Strikes, and SLOs at Scale

When Automation Fights Back (Jan 2018 – Mar 2019) Filed April 22, 2019 — the first anniversary of the TSB migration weekend. By 2018, most companies had automated their failover and recovery. The big postmortems this period are about that automation making the wrong call. The question stopped being “why did this break?” and became “why did our own self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018. A big-bang core-banking migration locked UK customers out of their accounts for weeks. The independent review that followed became required reading on cutover risk, and regulators started treating operational resilience as something they could enforce. GitHub, October 21, 2018. A 43-second network partition between the US East and West Coast datacenters led the orchestration software to promote a West Coast MySQL primary while the East Coast primary still held writes that hadn’t replicated. That’s a split-brain. GitHub chose data consistency over uptime, ran degraded for about 24 hours, and published a strong hour-by-hour writeup (github.blog). Microsoft Azure South Central US, September 2018. A lightning strike caused a cooling failure, hardware shut itself down to avoid damage, and the regional outage showed how many “global” Azure services (including Azure AD and the status page) secretly depended on one region. Google Cloud, July 2018. A global load-balancing config change briefly broke customers worldwide, reinforcing a growing theme: global control planes mean a global blast radius. Facebook, March 13, 2019. A server configuration change caused a roughly 14-hour outage of Facebook, Instagram, and WhatsApp, the longest in the company’s history at the time. Wells Fargo, February 2019. A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal Automated failover needs a plan for data. GitHub’s incident became the case study: failover that optimizes for uptime can quietly lose consistency. Postmortems started asking what the orchestrator actually does during a network partition. ...

April 22, 2019 · January 2018 – March 2019 · Retrospective

Patch Notes #153 — The Day We Saw It

Wednesday, humanity published a photograph of a black hole. M87*, fifty-five million light-years away: an orange ring of superheated doom around a shadow where physics stops answering emails. The Event Horizon Telescope is the engineering story of the decade hiding inside the science story of the decade: eight radio observatories across four continents synchronized into one Earth-sized virtual dish, generating so much data (petabytes) that it shipped by FedEx’d hard drives because sneakernet still beats backbone at that scale (the old speed-of-light constraint, now a feature). The correlation ran for two years; the imaging teams worked in deliberately isolated groups with different algorithms to make sure the ring wasn’t a shared artifact, blind-injection culture matured into blind-reconstruction culture. And the internet, correctly, made Katie Bouman’s hands-over-mouth photo the icon: behind every impossible image, a person watching their pipeline converge. ...

April 14, 2019

Patch Notes #152 — Streaming Everything: Games, TV, Credit

Google announced Stadia at GDC: games running entirely in their datacenters, streamed to any screen with Chrome, no console, no downloads, “up to 4K,” latency handled by (waves hands) edge presence and negative-latency prediction magic. The demo worked; the room split predictably between “consoles are over” and those of us who’ve spent careers learning what the speed of light does to interactive round-trips (the old CDN epiphany, now with 16ms frame budgets). My on-record position: the tech will mostly work in good conditions; the business is the hard part (whose games? what pricing? and Google’s product-graveyard reputation is now a real procurement objection: who buys a library that lives on a service with Google’s actuarial tables?). The idea is inevitable; this instance is questionable. Grade me in three years. ...

March 30, 2019

Patch Notes #151 — The Software Was Flying the Plane

Heavy entry. The world grounded the Boeing 737 MAX this week, every airline, every country, the FAA last among major regulators, after Ethiopian Airlines 302 crashed with the same signature as October’s Lion Air 610. 346 people across the two flights. The emerging picture, assembled from reporting and the Lion Air preliminary findings, centers on MCAS: flight-control software that automatically pushes the nose down based on angle-of-attack data, added to make a re-engined 60-year-old airframe handle like its predecessor, so airlines could skip simulator retraining. Per reporting: it took input from a single sensor (no voting, no cross-check), could re-engage repeatedly against pilot trim inputs, and wasn’t described in the flight manual because the certification strategy depended on the differences not counting as differences. ...

March 15, 2019

Patch Notes #150 — The City That Said No Thanks

Entry 150. The HQ2 file reopened exactly as flagged: Amazon cancelled the New York half of HQ2 on Valentine’s Day, 25,000 promised jobs withdrawn after sustained opposition to the ~$3B incentive package from local officials and organizers who asked, reasonably, why the world’s most valuable retailer needed a subsidy to hire in the largest talent pool on the continent. The postmortem splits cleanly by prior: “activists cost NYC 25k jobs” vs. “corporate site-selection-as-auction finally met a counterparty with leverage.” My file keeps to the mechanism design: the RFP-as-reconnaissance model works until one bidder audits the terms in public, and then the whole auction’s information asymmetry collapses at once. Virginia keeps its half, quietly, which tells you which negotiation style the machine prefers. ...

February 28, 2019

Patch Notes #149 — Ten Million People at a Concert That Wasn't There

The most important live event of the fortnight had no venue: Fortnite held an in-game Marshmello concert, ten-plus million concurrent players attending a virtual show, synchronized worldwide, with physics off and dance emotes mandatory. I attended, professionally (the group chat’s framed receipts obligated attendance). It was genuinely astonishing: not a video in a game, but a shared synchronized experience rendered live for a stadium the size of a mid-tier nation. The infrastructure implications alone (geospatial partitioning, now with a single global hot event on purpose) deserve a conference track, and the cultural implication deserves the decade: the kids’ third place isn’t the mall or the server room, it’s the lobby of a battle royale. Every “metaverse” pitch deck for the next five years just got its slide one, and for once the slide is real. ...

February 13, 2019