Patch Notes #154 — Cathedrals, Backups, and Portals

Notre-Dame burned the evening after #153 posted — the spire falling live on every feed, 850 years of accumulated civilization nearly lost to (per early reports) renovation-work ignition, saved at the last defensible line by firefighters who triaged like incident commanders: the towers hold or everything falls, so the towers get everything. Two files for the archive. One: the RELIC CHAIN worked — the crown of thorns and the rose windows survived because centuries of custodians maintained evacuation plans for objects, rehearsed (#135) across generations; the cathedral had RUNBOOKS, older than most nations. Two, the tech-adjacent thread everyone’s sharing: the building was comprehensively laser-scanned years ago (a billion-point cloud, by a scholar who died in 2018), and Ubisoft famously modeled it for Assassin’s Creed Unity — restoration will lean on DIGITAL TWINS made for scholarship and entertainment. Backups you didn’t know were backups (#007’s rumor clause, inverted and glorious): every faithful copy of a thing is a disaster-recovery asset, whatever it was made for. ...

April 29, 2019

Patch Notes #153 — The Day We Saw It

Wednesday, humanity published a photograph of a BLACK HOLE. M87*, fifty-five million light-years away: an orange ring of superheated doom around a shadow where physics stops answering emails. The Event Horizon Telescope is the engineering story of the decade hiding inside the science story of the decade — eight radio observatories across four continents synchronized into one EARTH-SIZED virtual dish, generating so much data (petabytes) that it shipped by FEDEX’D HARD DRIVES because sneakernet still beats backbone at that scale (#017’s speed-of-light constraint, now a feature). The correlation ran for two years; the imaging teams worked in DELIBERATELY ISOLATED groups with different algorithms to make sure the ring wasn’t a shared artifact — blind-injection culture (#076) matured into blind-RECONSTRUCTION culture. And the internet, correctly, made Katie Bouman’s hands-over-mouth photo the icon: behind every impossible image, a person watching their pipeline converge. ...

April 14, 2019

One Regex and a Pandemic: Global Blast Radius Meets Global Load

One Regex and a Pandemic (Apr 2019 – Jun 2020) This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth. The incidents that defined the period Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage. Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow. Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream. Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company. Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern. COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test. What the postmortems reveal 1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews. ...

April 1, 2019 · April 2019 – June 2020 · Retrospective

Patch Notes #152 — Streaming Everything: Games, TV, Credit

Google announced STADIA at GDC: games running entirely in their datacenters, streamed to any screen with Chrome — no console, no downloads, “up to 4K,” latency handled by (waves hands) edge presence and negative-latency prediction magic. The demo worked; the room split predictably between “consoles are over” and those of us who’ve spent careers learning what the speed of light does to interactive round-trips (#017’s CDN epiphany, now with 16ms frame budgets). My on-record position: the tech will mostly work in good conditions; the BUSINESS is the hard part (whose games? what pricing? and Google’s product-graveyard reputation — #083, #141 — is now a real procurement objection: who buys a library that lives on a service with Google’s actuarial tables?). The idea is inevitable; this instance is questionable. Grade me in three years. ...

March 30, 2019

Patch Notes #151 — The Software Was Flying the Plane

Heavy entry. The world grounded the Boeing 737 MAX this week — every airline, every country, the FAA last among major regulators — after Ethiopian Airlines 302 crashed with the same signature as October’s Lion Air 610. 346 people across the two flights. The emerging picture, assembled from reporting and the Lion Air preliminary findings, centers on MCAS: flight-control software that automatically pushes the nose down based on angle-of-attack data — added to make a re-engined 60-year-old airframe handle like its predecessor, so airlines could skip simulator retraining. Per reporting: it took input from a SINGLE sensor (no voting, no cross-check), could re-engage repeatedly against pilot trim inputs, and wasn’t described in the flight manual because the certification strategy depended on the differences not counting as differences. ...

March 15, 2019

Patch Notes #150 — The City That Said No Thanks

Entry 150. The #143 HQ2 file reopened exactly as flagged: Amazon CANCELLED the New York half of HQ2 on Valentine’s Day — 25,000 promised jobs, withdrawn after sustained opposition to the ~$3B incentive package from local officials and organizers who asked, reasonably, why the world’s most valuable retailer needed a subsidy to hire in the largest talent pool on the continent. The postmortem splits cleanly by prior: “activists cost NYC 25k jobs” vs. “corporate site-selection-as-auction finally met a counterparty with leverage.” My file keeps to the mechanism design (#143): the RFP-as-reconnaissance model works until one bidder audits the terms IN PUBLIC — then the whole auction’s information asymmetry collapses at once. Virginia keeps its half, quietly, which tells you which negotiation style the machine prefers. ...

February 28, 2019

Patch Notes #149 — Ten Million People at a Concert That Wasn't There

The most important live event of the fortnight had no venue: Fortnite held an in-game MARSHMELLO CONCERT — ten-plus million concurrent players attending a virtual show, synchronized worldwide, with physics off and dance emotes mandatory. I attended, professionally (the group chat’s framed receipts, #136, obligated attendance). It was genuinely astonishing: not a video IN a game, but a shared synchronized experience rendered live for a stadium the size of a mid-tier NATION. The infrastructure implications alone (#086’s geospatial partitioning, now with a single global hot event ON PURPOSE) deserve a conference track, and the cultural implication deserves the decade: the kids’ third place isn’t the mall or the server room, it’s the lobby of a battle royale. Every “metaverse” pitch deck for the next five years just got its slide one, and for once the slide is REAL. ...

February 13, 2019

Patch Notes #148 — The Bug That Listened

Apple shipped the year’s most intimate bug: Group FaceTime calls could be made to transmit AUDIO FROM THE RECIPIENT’S PHONE BEFORE THEY ANSWERED — call someone, add yourself to the group, and their microphone goes live while their screen still says “incoming call.” Discovered, per reporting, by a TEENAGER trying to set up a game chat, whose mother then spent over a WEEK trying to report it through Apple’s channels before it went viral and Apple killed Group FaceTime server-side entirely. Two files updated: the composition file (#140 — the bug lived in the INTERACTION of group-call state machines, each state individually sane) and a new one I’m opening on VULNERABILITY INTAKE: if a diligent citizen with a critical remote-eavesdropping bug can’t reach you in a week, your security program has a 404 where its front door should be. We tested our own security@ inbox Thursday. Response time: 4 hours. Relieved. Also: now monitored. ...

January 29, 2019

Patch Notes #147 — Year Seven: Scope Creep

Year seven begins with the fork from #146 already resolving: I spent the fortnight writing the platform-team proposal — headcount, charter, the paved-road philosophy (#106) promoted from metaphor to org chart. If it lands, I’ll be a senior engineer who spends 60% of the time on systems whose users are OTHER ENGINEERS. Internal platforms are products with the most brutally honest customers alive: they sit ten feet away and they WILL file the feedback in person, at lunch, forever. Terrifying. Correct. More as the org decides. ...

January 14, 2019

Patch Notes #146 — Year Six Retrospective: The Year of Invoices

Entry 146. Six years, 146 fortnights, zero gaps. The streak enters its seventh production year with 99.999% author uptime (one entry written on a phone in an Oregon eclipse field, #113; the SLA held). 2018’s THESIS: everything got INVOICED. 2017 ran up the bills (#121); 2018 collected. Facebook paid for the feed (Cambridge Analytica #127, $119B in a day #136, the token heist #140). Musk paid per word (#140: $4.4M/token). TSB paid for the skipped rehearsal (#130). Fallout 76 paid for shipping the wrong vision competently (#143). Marriott paid for buying a breach sight-unseen (#144). And the whole industry paid its GDPR retrofit costs (#131) for a decade of data hoarding — the only invoice that came with a receipt I’m actually proud of (#129’s deletion pipeline; the lawyers’ forcing function produced my year’s best work, therapy pending). ...

December 30, 2018