When the Map Burns with the Territory: BGP Lockouts and Cascading Dependencies

When the Map Burns with the Territory (Oct 2021 – Dec 2022) Filed January 11, 2023 — finished, fittingly, on the morning the FAA’s NOTAM outage grounded US departures and opened the next chapter. The image that defines this window: Facebook engineers reportedly couldn’t badge into their own buildings, because the outage had also taken down the door systems. Again and again, the tools meant to fix an outage turned out to depend on the thing that was broken. ...

January 11, 2023 · October 2021 – December 2022 · Retrospective

Patch Notes #244 — Principal

The whisper resolved: as of the January cycle, I’m a Principal Engineer. Ten years, near enough to the fortnight, from git reflog. The panel asked what I’d do with the role; I said “fewer things, harder to see” and watched the one principal on the panel smile. That’s the honest job description. A junior’s output is code; a senior’s is decisions; a staff’s is systems of decisions; a principal’s, near as I can tell from three weeks in, is the questions the org asks by default, the design-review prompts, the pre-mortem templates, the “who resigns over this?” that runs without me in the room. You know you’ve done principal work when something goes right two quarters later in a meeting you weren’t invited to. Compensation for the invisibility: this archive, which has been practicing exactly that accounting since 2013. ...

January 8, 2023

Patch Notes #243 — Year Ten Retrospective: The Final

The greatest World Cup final ever played. December 18th: Argentina 3, France 3, Messi scoring twice in his last Cup match, Mbappé answering with a 97-second two-goal resurrection and a hat trick in a final (losing it anyway, sport’s cruelest asterisk since the “28-3” entry), extra time’s dueling miracles, and penalties crowning Messi at last: the one trophy, chased across five World Cups, that his God-tier career was somehow incomplete without, delivered at 35 in his declared farewell (the fourth-ring-weight doctrine receiving its maximum case: the ending is part of the work, and this ending was written by someone sentimental, per the old Jeter clause). The group chat did not survive intact; the German engineer conceded it was the best final he’d ever seen; the archive’s ten-year sports thread, longevity is the skill, closes its decade with its perfect proof standing on a podium in a bisht, holding the thing. ...

December 24, 2022

Patch Notes #242 — The Chat Window That Ate the Discourse

On November 30th, OpenAI released a “research preview” called ChatGPT, the earlier thread (GPT-3’s API demos) wrapped in a chat interface with RLHF post-training, and the fortnight since has been the fastest product adoption event in recorded history: a million users in five days, my non-tech family members sending me screenshots, and every group chat this archive feeds from conducting the same experiment simultaneously (write my performance review; explain my toddler’s sleep regression as a Nabokov passage; debug this regex, it did, the old backtracking monster, correctly, with an explanation). The staff-file assessment, written carefully because this one will be re-read: the interface is the revolution more than the model, GPT-3.5’s capabilities existed in API form for months (the four-day tools); wrapping them in dialogue with memory and refusal training converted a developer curiosity into a consumer epiphany, the Wordle lesson (distribution is design) at planetary scale. The failure modes are unchanged and now globally distributed (confident confabulation, fluency-as-failure-mode, meets a billion users who read fluency as authority; the citation-inventing lawyer incidents are pre-registered herewith), the academic-integrity panic has arrived on schedule (take-home essays met their classifier moment overnight), and our own office’s censused usage went from zero to “most engineers, daily” in ten days, boilerplate, tests, unfamiliar-API scaffolding, the fastest tool adoption I’ve witnessed in a career of platform evangelism (the funnel, self-executing; my job just changed and the file knows it). The quiet thread is no longer quiet, no longer a thread: as of this fortnight it is the loom. The 2023 file is pre-named. ...

December 9, 2022

Patch Notes #241 — Bankruptcy Filings and Group-Stage Miracles

The FTX collapse completed and exceeded its worst readings: Chapter 11 filed November 11th (~$8B customer shortfall confirmed in effect), SBF’s empire dissolved into the most damning first-day filing in bankruptcy history, the new CEO, John J. Ray III, the man who administered Enron’s corpse, wrote under oath: “Never in my career have I seen such a complete failure of corporate controls,” no board to speak of, no complete employee list, corporate funds buying Bahamian real estate in personal names, approvals via emoji in group chats, and (the detail this archive files as the decade’s governance epitaph) QuickBooks. A $32B financial institution, on QuickBooks. The adversarial-reviewer doctrine’s terminal case: venture’s finest wired nine figures into an entity with no cap table on file, because the founder wore the right shorts and funded the right causes, diligence replaced by vibes-at-scale one final, maximal time (the group-chat papers were the dress rehearsal). Criminal referrals are foregone; the contagion tree grows its final branches (BlockFi froze withdrawals within days and its bankruptcy filing is, per reporting, a formality in progress); and the file’s decade-long crypto thread receives its Mt. Gox-to-FTX symmetry with grim completeness: the industry’s biggest failure was never the cryptography, it was custody, governance, and the oldest sin, every single time (the ledger was distributed, the fraud was centralized). ...

November 24, 2022

Patch Notes #240 — The Fortnight the Ledgers Opened

The densest fourteen days in this archive’s ten years; filed in three ledgers, all still moving. Twitter: The deal closed October 27th (“the bird is freed”); within the week: the board dissolved, ~50% of staff laid off in a single day (email-at-dawn, access-revoked-first, the craftsmanship file’s exact anti-pattern executed at maximum scale and speed), advertiser pauses cascading (GM, United, the majors, the debt-vise thesis meeting revenue in freefall), and today the $8 blue-check launched into an instant impersonation carnival (the brand-parody parade began within hours; the file pre-registers that a blue-checked fake moves a real company’s stock price within the week — identity infrastructure repriced to $8 is about to discover that verification was load-bearing, the defaults doctrine’s most expensive demonstration yet). The earlier pre-registration (operational before ideological) is grading correct at horrifying velocity; the remaining engineers’ war stories will fill this file for years. ...

November 9, 2022

Patch Notes #239 — Escrow Week

Transition fortnight, the calm of wires moving. The Twitter close is days away (financing confirmed in escrow, the Delaware docket holstered but loaded; Musk has been tweeting renovation plans and vibes, and the archive braces for whatever pun-based performance art the closing week will surely bring), the founder-acquisition experiment begins in earnest within the week, and the file spends its waiting room on the structural read it will grade against: the acquisition closes into the worst ad market in a decade (the old macro), with $13B of leveraged-buyout debt now strapped to a company that has rarely been profitable unlevered, meaning the ideological experiment everyone’s watching sits atop a financial vise almost nobody’s pricing: interest service alone will demand either revenue invention or cost demolition at speeds no social platform has survived gracefully (the earlier pre-registration, operational failures before ideological ones, now has a mechanism attached: the debt is the operations story). Fifteen days from now this entry’s restraint will look either wise or quaint; the archive has stopped pretending to know which in advance, which is itself the ten-year lesson (the calibration file, humbled and better for it). ...

October 25, 2022

Patch Notes #238 — The Prompt Is the Product

A deliberately image-heavy entry, because the fortnight’s real story is visual: the generative-AI wave (the humming crescendo) has broken into full public view, DALL-E 2 dropped its waitlist (Sep 28), Stable Diffusion has been fully open-weights for a month (the August release the file now recognizes as the inflection: within weeks the internet built UIs, plugins, fine-tunes, and workflows no lab roadmap imagined, open weights turned a product into an ecosystem, the left-pad lesson running in reverse: a thousand dependencies blooming from one artifact), Midjourney’s Discord is generating a museum’s worth of imagery daily, and an AI-generated piece won the Colorado State Fair’s digital-art competition (the artist-outrage discourse arriving exactly on the Move-37 schedule: first it’s a trick, then it’s a tool, then it’s a threat, then it’s the field). The staff-file’s assessment, filed with both hands: the capability curve is genuine and compounding (text-to-image crossed “useful” this year the way text crossed “startling” in 2020); the labor questions (training data provenance, style mimicry, the illustrators watching their portfolios become gradients) are real, unresolved, and heading for courts (pre-registered: the copyright litigation defining this decade files within months); and the interface lesson is the sleeper, “prompt engineering” is a genuinely new craft layer, natural language as the API (the few-shot lesson maturing into a job description). Our own ticket-summarizer gained an image-generation sibling this sprint (marketing mockups; four days, one engineer, again, the capability-to-effort ratio, the file repeats, is the entire story of the coming year). ...

October 10, 2022

Patch Notes #237 — The Merge Merged

It worked. September 15th, 06:42 UTC: Ethereum’s consensus layer hot-swapped from proof-of-work to proof-of-stake mid-flight (the countdown resolving clean), block 15,537,393 mined, block 15,537,394 validated, no downtime, no chain split of consequence, energy consumption down the projected ~99.95% overnight (the single largest voluntary emissions reduction by any industry, ever, executed as a software deploy). The file’s engineering awe, on the record: this was years of client rehearsal, shadow forks, testnet merges (Ropsten, Sepolia, Goerli as staging environments for a $200B production system), multiple independent client implementations cross-validating so no single codebase’s bug could kill consensus (the monoculture-is-the-vulnerability doctrine, designed against from day one), the rehearsal gospel performed at the highest financial stakes in open-source history. Whatever the file’s positions on the asset class (extensive, mixed, see: entire archive), the migration enters its all-time top shelf next to Webb and the M1 transition: 2022, year of the impeccable cutover, in the middle of the industry’s messiest everything-else. (The miners’ severance: a forked ETHW chain nobody much wanted, even perfect migrations strand someone; the deprecation had no clean answer for the GPUs, and used-card prices are the market’s memorial.) ...

September 25, 2022

Patch Notes #236 — Three More Rounds of Magic

Serena’s farewell exceeded every script: three rounds of vintage, stadium-shaking tennis, including a second-round upset of the world #2 that made Ashe sound like a launch pad, before falling in three sets to Tomljanović on September 2nd, at 40, having turned a retirement announcement into the most-watched tennis matches in US history. The ending held everything the file admires: peak-end discipline (she left on a run, not a decline), the crowd’s twenty-minute ovation, and the post-match “I wouldn’t be Serena if there wasn’t Venus,” the footnotes-build-everything doctrine delivered as a sister’s tribute. The archive’s sports thread files her with Federer and the Nadal entry under its one repeating thesis: longevity is the skill, and the ending is part of the work. ...

September 10, 2022