Patch Notes #125 — Turin Special, Starman, and the Grades

Grading #124’s predictions, as promised. Went 0-for-2 in the best possible way. SPURS DREW IN TURIN. Not Juve-by-2 — a 2-2 shootout where Tottenham came back from 2-0 down, capped by a tactical trick: a backup midfielder dropping deep to feed Kane on a decoy run the manager installed for exactly this moment. Champions League-shaking audacity from the understudy. The away end in Turin was jubilant, gloriously. Backup systems, properly drilled, can outperform the primary (#045’s MTTR sermon in cleats). ...

February 18, 2018

Patch Notes #124 — Eve of Everything

A rare cliffhanger entry — this fortnight is all EVES, and I’m logging predictions before outcomes so the record can grade me. Next week: Champions League Round of 16, Tottenham vs Juventus. Spurs are starting their backup defender against the Italian defensive dynasty (#100); the fanbase has braced itself in anticipatory self-defense. Prediction: Juve by 2, with maximum pain. (I want Spurs to advance. Wanting is not forecasting.) Tuesday: SpaceX attempts the first FALCON HEAVY launch — twenty-seven engines, three boosters, the most powerful rocket since the Saturn V, with Elon’s personal Tesla Roadster as the dummy payload because the payload-mass simulator rules don’t say it CAN’T be a car. Musk himself puts odds at coin-flip and says success is “not blowing up the pad.” The two side boosters are supposed to land BACK, together, synchronized. Prediction: it flies, one booster sticks. (The #073 kid who cried at one landing is emotionally unprepared for two.) ...

February 3, 2018

Patch Notes #123 — This Is Not a Drill (It Was a Drill)

Last Saturday morning, every phone in Hawaii received: “BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.” For THIRTY-EIGHT MINUTES, it stood. People put children in storm drains. Said goodbye. And the cause, when it emerged, was the most professionally humbling artifact of the year: an employee running a shift-change drill selected the wrong item from a dropdown menu — the real-alert option living adjacent to the test option, same list, similar labels, with a confirmation dialog that (like all confirmation dialogs) had been trained into muscle-memory clickthrough. No cancel template existed; one had to be COMPOSED, mid-crisis, over 38 minutes. ...

January 19, 2018

Patch Notes #122 — The CPU Was Lying the Whole Time

Year six opens with the most unsettling vulnerability disclosure I’ve ever read: MELTDOWN and SPECTRE. Not bugs in software — bugs in the IDEA of modern CPUs. Speculative execution, the trick where processors guess ahead to go fast, turns out to leak secrets through timing side channels. Meltdown lets a process read kernel memory; Spectre tricks other processes into leaking their own; between them, essentially every Intel chip since the 90s and most others are affected. The fix costs PERFORMANCE (the cloud providers are rebooting the entire planet’s fleet this week — imagine THAT change-management ticket), and Spectre-class attacks will haunt chip design for a decade because the flaw is load-bearing: the speed we’ve enjoyed since 1995 was partially borrowed against an invariant nobody wrote down. ...

January 4, 2018

When Automation Fights Back: Split Brains, Lightning Strikes, and SLOs at Scale

When Automation Fights Back (Jan 2018 – Mar 2019) By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from “why did the component fail?” to “why did our self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018 — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain. GitHub, October 21, 2018 — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. Split-brain. GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (github.blog). Microsoft Azure South Central US, September 2018 — A lightning strike caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many “global” Azure services (including Azure AD and the status portal) had hidden dependencies on one region. Google Cloud, July 2018 — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius. Facebook, March 13, 2019 — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a server configuration change — at the time the longest outage in the company’s history. Wells Fargo, February 2019 — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal 1. Automated failover needs a theory of data. GitHub’s incident became the case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking “what does our orchestrator do during a partition?” — a Jepsen-style question applied to ops tooling. ...

January 1, 2018 · January 2018 – March 2019 · Retrospective

Patch Notes #121 — Year Five Retrospective: Peak Something

Entry 121, year five complete, streak intact. This one’s dated the 20th because the 25th belongs to family and the 30th to Zelda — seniority is knowing your own load limits (#111). The closing fortnight compressed the whole year: net neutrality was repealed on the 14th as scheduled (#119’s slow-boil forecast now on the clock); Bitcoin kissed ~$19,783 on the 17th and is already wobbling (if that was the top, let the record show the top smelled like my barber’s price targets); and The Last Jedi opened to a fascinating split — critics elated, a vocal fan-segment furious — which mostly taught me that beloved legacy systems can’t be refactored without someone filing a grievance about the original architecture. (I liked it. The Luke arc IS the brave design choice.) ...

December 20, 2017

Patch Notes #120 — Digital Cats Ate the World Computer

The most 2017 sentence ever written, and I get to write it: trading of CARTOON CATS has congested the Ethereum network. CryptoKitties — collectible, breedable, blockchain-native cats — launched last week and immediately became the largest consumer of gas on the “world computer,” at points comprising 15-20%+ of ALL network traffic, backing up transactions globally and spiking fees for everyone trying to do, you know, finance. People have paid over $100,000 for a single virtual cat. The serious take hiding in the absurdity: this is the first ORGANIC consumer dapp product-market fit, and it instantly hit the scaling wall — Ethereum does ~15 transactions per second, total, planet-wide, and one viral toy saturated it (the whole chain is one hot partition, #086, by DESIGN). Every scaling roadmap conversation (sharding, layer-2, state channels) just got its forcing function, and its mascot. Thundering herds (drink) now come in kitten form. ...

December 5, 2017

Patch Notes #119 — Trucks, Coins, and Countdown Clocks

The FCC formally scheduled the net-neutrality repeal vote for December 14th, and the outcome is not in suspense — the #053 victory gets rolled back on a party-line vote almost three years to the season. Logging my calibrated take alongside the outrage: the apocalypse scenarios (per-site tolls tomorrow!) are overwrought on day one, because ISPs read headlines too; the REAL erosion will be slow, boring, and bundled — zero-rating here, interconnect leverage there, each individually defensible, compounding like the frog’s bathwater. Infrastructure rarely fails loudly when it can fail gradually (this is the whole genre of my job). The courts and states get the next moves. ...

November 20, 2017

Patch Notes #118 — Mexico City Statements

HAMILTON won the F1 World Championship — his fourth title, in a dramatic Mexican GP that included a first-lap collision with Vettel that dropped him to the back of the grid and aged every Mercedes engineer a fiscal year. For Hamilton, years after his first title with allegations of rookie struggles, the victory cemented him among the legends. The team-build story is pure engineering-org porn, too: Mercedes built a data-first power unit system that experts predicted would dominate the turbo-hybrid era, and they shipped exactly on the roadmap. The most audacious long-term engine refactor in motorsport, delivered to spec. (Retention is architecture, #069; so is planned demolition.) ...

November 5, 2017

Patch Notes #117 — Zero

DeepMind published AlphaGo ZERO this week, and the result reorganized my priors more than the original (#078) did. The first AlphaGo learned from 30 million human expert moves, then surpassed us. Zero learned from NOTHING — the rules, a board, and self-play. Three days: it beat the version that beat Lee Sedol, 100 games to 0. Forty days: it beat every AlphaGo ever built. Along the way it REDISCOVERED centuries of human joseki (opening theory), used them for a while, and then DISCARDED some in favor of moves we’d never found in three thousand years. Human knowledge, it turns out, wasn’t the ladder — it was scaffolding, and load-bearing bias. The bitter arithmetic: our accumulated expertise was worth about three days of self-play. I’ve retold Move 37 (#078) as “the machine surprised us”; Zero’s lesson is stranger — WE were the constraint being optimized away. Centaurs-over-engines (#078) needs an asterisk I don’t have words for yet. Ask me again in ten years, again. ...

October 21, 2017