Patch Notes #149 — Ten Million People at a Concert That Wasn't There

The most important live event of the fortnight had no venue: Fortnite held an in-game Marshmello concert, ten-plus million concurrent players attending a virtual show, synchronized worldwide, with physics off and dance emotes mandatory. I attended, professionally (the group chat’s framed receipts obligated attendance). It was genuinely astonishing: not a video in a game, but a shared synchronized experience rendered live for a stadium the size of a mid-tier nation. The infrastructure implications alone (geospatial partitioning, now with a single global hot event on purpose) deserve a conference track, and the cultural implication deserves the decade: the kids’ third place isn’t the mall or the server room, it’s the lobby of a battle royale. Every “metaverse” pitch deck for the next five years just got its slide one, and for once the slide is real. ...

February 13, 2019

Patch Notes #148 — The Bug That Listened

Apple shipped the year’s most intimate bug: Group FaceTime calls could be made to transmit audio from the recipient’s phone before they answered. Call someone, add yourself to the group, and their microphone goes live while their screen still says “incoming call.” Discovered, per reporting, by a teenager trying to set up a game chat, whose mother then spent over a week trying to report it through Apple’s channels before it went viral and Apple killed Group FaceTime server-side entirely. Two files updated: the composition file (the bug lived in the interaction of group-call state machines, each state individually sane) and a new one I’m opening on vulnerability intake: if a diligent citizen with a critical remote-eavesdropping bug can’t reach you in a week, your security program has a 404 where its front door should be. We tested our own security@ inbox Thursday. Response time: 4 hours. Relieved. Also: now monitored. ...

January 29, 2019

Patch Notes #147 — Year Seven: Scope Creep

Year seven begins with last entry’s fork already resolving: I spent the fortnight writing the platform-team proposal, headcount, charter, the paved-road philosophy promoted from metaphor to org chart. If it lands, I’ll be a senior engineer who spends 60% of the time on systems whose users are other engineers. Internal platforms are products with the most brutally honest customers alive: they sit ten feet away and they will file the feedback in person, at lunch, forever. Terrifying. Correct. More as the org decides. ...

January 14, 2019

Patch Notes #146 — Year Six Retrospective: The Year of Invoices

Entry 146. Six years, 146 fortnights, zero gaps. The streak enters its seventh production year with 99.999% author uptime (one entry written on a phone in an Oregon eclipse field; the SLA held). 2018’s thesis: everything got invoiced. 2017 ran up the bills; 2018 collected. Facebook paid for the feed (Cambridge Analytica, $119B in a day, the token heist). Musk paid per word ($4.4M/token). TSB paid for the skipped rehearsal. Fallout 76 paid for shipping the wrong vision competently. Marriott paid for buying a breach sight-unseen. And the whole industry paid its GDPR retrofit costs for a decade of data hoarding, the only invoice that came with a receipt I’m actually proud of (the deletion pipeline; the lawyers’ forcing function produced my year’s best work, therapy pending). ...

December 30, 2018

Patch Notes #145 — The Purge and the Spaceplane

Tumblr’s adult-content ban takes effect Monday, announced two weeks ago (proximate cause: a child-safety app-store removal; structural cause: Verizon-owned Tumblr’s ads business needing brand safety), and executed via an ML content classifier whose false positives have become a genre unto themselves: flagged sand dunes, flagged classical sculpture, flagged Tumblr’s own announcement post. Two lessons for the file. One: content moderation at scale is classifier deployment at scale, and shipping a high-stakes model with that precision profile is the ML equivalent of the TSB cutover; the confusion matrix is the product. Two, the bigger one: communities are load-bearing (the Reddit lesson) and this is its corollary: a platform purging its core community’s content is a migration event for that community, and the diaspora (Twitter, bespoke sites, Discord) is already routing around the damage. Platforms are temporary; archives are personal. Export your things. ...

December 15, 2018

Patch Notes #144 — Five Hundred Million Check-Ins

Marriott disclosed today: the Starwood reservation database was breached for four years, since 2014, predating Marriott’s own acquisition of Starwood, exposing up to 500 million guests: passports, travel histories, the occasional unencrypted card. Two archive threads converge with a clang. First, the M&A angle: Marriott bought this breach in 2016 and ran it unknowingly for two years. Due diligence audits the books, the brand, the real estate, and treats the IT estate as furniture; “you acquire their attackers too” belongs in every deal memo (Yahoo’s discount was the precedent; a breach literally repriced that acquisition, and the industry filed it under “Yahoo problems”). Second, the duration: four years of dwell time. Breach detection lag is the metric nobody dashboards; prevention gets the budget, detection gets a log-retention policy written by the finance team. (The ceiling rule applies; expect the passport count to firm upward.) ...

November 30, 2018

Patch Notes #143 — Excelsior

Stan Lee died Monday at 95, and the internet performed the full global grief ritual (the Prince protocol) for the man whose collaborative mythology (with Kirby, with Ditko; the footnotes built half of everything, same week’s lesson) became this century’s shared narrative operating system. The MCU is architecture: twenty films on a common continuity substrate, the boldest long-term integration project in entertainment (the Infinity War snap was its TSB-scale cutover, executed clean). Excelsior, and credit the co-authors. ...

November 15, 2018

Patch Notes #142 — 43 Seconds, 24 Hours

The GitHub outage of October 21st got its postmortem this week, and it’s an instant classic of the genre, the kind I’ll assign to juniors like homework: a routine maintenance event caused a 43-second network partition between GitHub’s East and West coast datacenters. Forty-three seconds. The failover automation, doing exactly its job, promoted a West Coast MySQL primary while the East Coast primary held seconds of writes that had never replicated. Split-brain: two databases, each believing itself the truth. And here’s the decision that makes it teaching material: GitHub chose consistency over uptime, running degraded for 24+ hours to reconcile data rather than serve wrong answers fast. The postmortem says so, explicitly, trade-off by trade-off. That’s the maturity frontier: not preventing all failure (43 seconds of network weather defeated a world-class team) but choosing your failure mode in advance and documenting the choice like an adult (rehearsed audacity, ops edition). Our own failover’s partition behavior is now a scheduled game-day. I checked. Nobody knew the answer. That’s the finding. ...

October 31, 2018

Patch Notes #141 — Ghosts, Giants, and Grain-of-Rice Chips

Paul Allen died Monday, 65, lymphoma, the other Microsoft founder, the one who named it, who talked Gates into the BASIC bet that started everything, then spent his post-Microsoft decades funding brain science, rock museums, sports franchises, and rocket planes. The industry’s origin generation is becoming its memorial generation, and the eulogies this week were a reminder that “co-founder” histories get flattened by the survivor’s spotlight. Read the footnotes; the footnotes built half of everything. ...

October 16, 2018

Patch Notes #140 — The Tweet's Invoice and the Token Heist

The “funding secured” saga reached settlement in a single whiplash week: the SEC sued Musk for securities fraud Thursday, seeking to bar him from running any public company; by Saturday a deal was inked, Musk steps down as chairman (keeps CEO), pays $20M (Tesla pays another $20M), and, in the detail history will remember, Tesla must implement oversight of his tweets. A court-mandated code review for a CEO’s social media. The nine words cost $40M and a chairmanship: roughly $4.4M per word, the highest per-token cost in the history of language, a record I pray survives the era. Guardrails-for-the-manic-day: now case law. ...

October 1, 2018