Patch Notes #145 — The Purge and the Spaceplane

Tumblr’s adult-content ban takes effect Monday — announced two weeks ago (proximate cause: a child-safety app-store removal; structural cause: Verizon-owned Tumblr’s ads business needing brand safety), and executed via an ML content classifier whose false positives have become a genre unto themselves: flagged sand dunes, flagged classical sculpture, flagged Tumblr’s OWN announcement post. Two lessons for the file. One: content moderation at scale is classifier deployment at scale, and shipping a high-stakes model with THAT precision profile is the ML equivalent of the TSB cutover (#130) — the confusion matrix IS the product. Two, the bigger one: communities are load-bearing (#061’s Reddit lesson) and this is its corollary — a platform purging its core community’s content is a MIGRATION EVENT for that community, and the diaspora (Twitter, bespoke sites, Discord) is already routing around the damage. Platforms are temporary; archives are personal (#087). Export your things. ...

December 15, 2018

Patch Notes #144 — Five Hundred Million Check-Ins

Marriott disclosed today: the Starwood reservation database was breached for FOUR YEARS — since 2014, predating Marriott’s own acquisition of Starwood — exposing up to 500 million guests: passports, travel histories, the occasional unencrypted card. Two archive threads converge with a clang. First, the M&A angle: Marriott BOUGHT this breach in 2016 and ran it unknowingly for two years — due diligence audits the books, the brand, the real estate, and treats the IT estate as furniture; “you acquire their attackers too” belongs in every deal memo (#087’s Yahoo discount was THE precedent — a breach literally repriced that acquisition, and the industry filed it under ‘Yahoo problems’). Second, the duration: four YEARS of dwell time. Breach detection lag is the metric nobody dashboards — prevention gets the budget, detection gets a log-retention policy written by the finance team. (#116’s ceiling rule applies; expect the passport count to firm upward.) ...

November 30, 2018

Patch Notes #143 — Excelsior

Stan Lee died Monday at 95, and the internet performed the full global grief ritual (#081’s Prince protocol) for the man whose collaborative mythology — WITH Kirby, WITH Ditko; the footnotes built half of everything, #141, same week’s lesson — became this century’s shared narrative operating system. The MCU is architecture: twenty films on a common continuity substrate, the boldest long-term integration project in entertainment (#130’s Infinity War snap was its TSB-scale cutover, executed clean). Excelsior, and credit the co-authors. ...

November 15, 2018

Patch Notes #142 — 43 Seconds, 24 Hours

The GitHub outage of October 21st got its postmortem this week, and it’s an instant classic of the genre — the kind I’ll assign to juniors like homework: a routine maintenance event caused a 43-SECOND network partition between GitHub’s East and West coast datacenters. Forty-three seconds. The failover automation, doing exactly its job, promoted a West Coast MySQL primary — while the East Coast primary held seconds of writes that had never replicated. Split-brain: two databases, each believing itself the truth. And here’s the decision that makes it teaching material: GitHub chose CONSISTENCY over uptime — running degraded for 24+ hours to reconcile data rather than serve wrong answers fast. The postmortem SAYS so, explicitly, trade-off by trade-off. That’s the maturity frontier: not preventing all failure (43 seconds of network weather defeated a world-class team) but choosing your failure MODE in advance and documenting the choice like an adult (#125’s rehearsed audacity, ops edition). Our own failover’s partition behavior is now a scheduled game-day. I checked. Nobody knew the answer. THAT’S the finding. ...

October 31, 2018

Patch Notes #141 — Ghosts, Giants, and Grain-of-Rice Chips

Paul Allen died Monday — 65, lymphoma, the OTHER Microsoft founder, the one who named it, who talked Gates into the BASIC bet that started everything, then spent his post-Microsoft decades funding brain science, rock museums, sports franchises, and rocket planes. The industry’s origin generation is becoming its memorial generation, and the eulogies this week were a reminder that “co-founder” histories get flattened by the survivor’s spotlight. Read the footnotes; the footnotes built half of everything. ...

October 16, 2018

Patch Notes #140 — The Tweet's Invoice and the Token Heist

The “funding secured” saga (#137, #139) reached settlement in a single whiplash week: the SEC sued Musk for securities fraud Thursday, seeking to bar him from running ANY public company; by Saturday a deal was inked — Musk steps down as CHAIRMAN (keeps CEO), pays $20M (Tesla pays another $20M), and, in the detail history will remember, Tesla must implement OVERSIGHT OF HIS TWEETS. A court-mandated code review for a CEO’s social media. The nine words cost $40M and a chairmanship: roughly $4.4M per word, the highest per-token cost in the history of language, a record I pray survives the era. Guardrails-for-the-manic-day (#137): now case law. ...

October 1, 2018

Patch Notes #139 — Double Standards and Double Cameras

The US Open women’s final became the sport’s most argued-about match in years: Serena, chasing a record 24th slam post-maternity, received three code violations (coaching, racket abuse, verbal abuse — the last for calling the umpire a “thief”) while 20-year-old Naomi Osaka played flawless, devastating tennis on the other side. The trophy ceremony — Osaka’s first slam, WON on merit, received in tears under a booing stadium — was the worst-orchestrated victory moment I’ve ever watched. Two things demand simultaneous filing, which the discourse refuses: Osaka was the better player and deserved a clean coronation; and the enforcement-consistency question Serena raised (male players’ documented latitude for far worse) is real and measurable. Selective enforcement of rarely-enforced rules is the referee equivalent of the flaky test (#081) — technically “correct” on any single invocation, corrosive to trust across the suite. Fix the suite or lose the signal. ...

September 16, 2018

Patch Notes #138 — Hiring Season Dispatch

Slow news fortnight (post-funding-secured, pre-fall-launches), so a working dispatch from the interview trenches, where I’ve spent this quarter as loop lead for our next eng cohort — the #089 work-sample experiment, two years matured, with data worth reporting. What’s held: work-sample auditions beat whiteboard algorithms by every measure we track — signal quality, candidate experience scores, and (the one that matters) six-month on-the-job correlation. What’s surprised me: the STRONGEST signal in the whole loop is the debugging session — we hand candidates a real (sanitized) broken service from our incident archive and watch them navigate. Not solve; NAVIGATE. Do they form hypotheses or thrash? Read logs or guess? Say “I don’t know, here’s how I’d find out”? Five years of this blog is essentially that skill journaled (#001’s reflog to #135’s rehearsals), and watching candidates perform it live has taught me it’s rarer than algorithmic fluency and vastly less practiced. The industry drills LeetCode; the job is 70% forensics. We interview for the drill. ...

September 1, 2018

Patch Notes #137 — Funding Secured

On August 7th, mid-trading-day, Elon Musk tweeted: “Am considering taking Tesla private at $420. Funding secured.” Nine words, one weed joke of a price, and a claim — SECURED — that moved billions in market cap within minutes. The ten days since: the funding was, per subsequent reporting, more “discussed” than “secured” (Saudi fund talks in early stages); the board scrambled; the SEC has reportedly subpoenaed; and the whole affair is a live-fire seminar in why material statements from CEOs have rules that don’t care about the medium. A tweet is a press release with worse review tooling — no legal sign-off, no edit button (in 2018), maximum velocity. My compliance-adjacent year (#131) has made me boring about this: the guardrail isn’t for the honest day, it’s for the manic one (#123, dropdown menus for executives). This one runs for months; logging the opening move. ...

August 17, 2018

Patch Notes #136 — Thirteen Digits

Apple became the first TRILLION-dollar company today. $1,000,000,000,000 — the garage company, the 90-days-from-bankruptcy company (1997), the courage company (#090), first to thirteen digits. The arc from Jobs’ return to here is the greatest turnaround in business history and the least repeatable: it required a founder-artist, a supply-chain grandmaster (now CEO), and a product that put a computer against every human femur on Earth. The office ran a sweepstake on which company hits $2T first and WHEN; my entry says Apple again, 2022, and writing it here guarantees accountability and probable ridicule. ...

August 2, 2018