Patch Notes #225 — Bridges, Breaches, and the Disclosure Gap

The Lapsus$ arc resolved on schedule and on thesis: London police arrested seven people aged 16-21 (the “study group” pre-registration grades correct), but not before the crew’s Okta breach detonated into the fortnight’s real lesson, not the intrusion (a support contractor’s laptop, January, contained scope) but the disclosure: Okta, the identity provider for thousands of companies, the literal login layer of the enterprise internet, knew in January, concluded limited impact, and said nothing until Lapsus$ posted screenshots in March, forcing a staggered, defensive drip of statements that converted a contained incident into a trust crisis. The file’s doctrine, engraved by now (the breach-incentives file, the Google+ calculus): for infrastructure-of-trust vendors, the disclosure is the product, customers weren’t asking “were you breached?” but “will you tell us when you are?”, and the answer arrived empirically. Our own vendor-review checklist gained a question this sprint: “describe your last disclosure decision, with timeline.” The answers are more predictive than any SOC 2. ...

March 29, 2022

Patch Notes #224 — The Teenagers Inside the Fortune 500

The fortnight’s security story is Lapsus$, a crew hacking its way through Nvidia (source code and certificate-signing material out), Samsung (Galaxy source), Ubisoft, and reportedly more, with more names rumored to be on the tour, and the file’s fascination is their method, which is barely “hacking” in the movie sense at all: SIM-swaps, insider recruitment (openly advertising payment for employee credentials on Telegram), MFA-fatigue bombing (spam push notifications until a tired human taps approve), and help-desk social engineering. No zero-days; just the human layer, industrialized (the Twitter-teens file, now with an org chart and a marketing channel). They leak like performers, polls asking followers which stolen source to drop next, and the operational-security errors suggest actual teenagers, which subsequent arrests may confirm (the file pre-registers: the most effective threat actor of the quarter is probably a study group). The defensive translation, urgent and unglamorous: MFA is not MFA, push-approval fatigue is a designed-in vulnerability (number-matching and hardware keys exist; deploy them), help desks need verification runbooks with teeth, and “insider threat” now includes “employee recruited by DM for $20k” (the org-chart-as-attack-surface, with a price list). ...

March 14, 2022

Patch Notes #223 — War, Rendered in Every Layer

Russia invaded Ukraine on February 24th. The archive’s lane feels small against the human scale, cities shelled, a million refugees moving, a European land war in the era of TikTok, and the file proceeds humbly, logging the layers where its competence applies, because every one of its ten-year threads is suddenly live ordnance: sanctions as infrastructure (SWIFT disconnection deployed within days, the payment-rails-as-sovereignty doctrine, at nation scale; central-bank reserve freezes teaching every treasury on Earth that foreign-held assets are conditional); platform geopolitics (Meta, Google, and Apple restricting Russian state media and services under simultaneous pressure from EU regulators and Russian censors, the values-pricing file with no neutral configuration available); Starlink (terminals shipped to Ukraine within days of a tweeted request, the Tonga thread’s civil-infrastructure moment arriving in a war zone; commercial satellite constellations are now strategic assets, with everything that implies about their owners); cyber (the predicted apocalypse arriving as wiper malware and the Viasat modem attack, significant, targeted, and notably not the grid-collapse scenario; while a volunteer “IT Army” DDoSes Russian targets, dissolving the combatant/civilian line in ways international law hasn’t versioned for); and information (Ukraine’s government running the most effective wartime comms operation ever conducted, president-on-a-phone-camera defiance versus a state media apparatus, asymmetric warfare’s newest theater is the feed, the outrage-optimizer conscripted by every side). ...

February 27, 2022

Patch Notes #222 — The Quarter Meta Fell to Earth

Meta reported earnings February 3rd and the market performed the largest single-day value deletion in history: -26%, roughly $230 billion erased (dwarfing the record it already held), triggered by one number the archive has waited a decade to see: Facebook’s daily active users declined quarter-over-quarter for the first time ever. The growth curve that financed everything (the $50B scandal-shrug, the testimony armor) showed its first negative derivative, compounded by Apple’s App Tracking Transparency (that old privacy architecture now costing Meta a reported $10B/year in ad targeting, platform power exercised via a permissions dialog, the defaults doctrine weaponized between giants) and TikTok’s attention conquest (the algorithmic-feed disruption arriving from the flank nobody’s antitrust filings mapped). The renaming’s timing thesis upgrades from cynical to actuarial: the pivot was announced one quarter ahead of the curve it was fleeing. The metaverse burn ($10B/year) now reads as a company-scale bet that the next platform can be owned since the current one is peaking, history’s most expensive “the org chart is the last to know” hedge. ...

February 12, 2022

Patch Notes #221 — Sixty-Nine Billion Dollars of Content

Microsoft announced it’s buying Activision Blizzard for $68.7 billion, all cash, the largest gaming acquisition in history by a factor of three, the largest Microsoft acquisition ever (2.5x LinkedIn), and a deal whose every layer earns file space: the strategy layer (Game Pass as the Netflix-of-games needs a content moat; Call of Duty, Warcraft, and, the sleeper asset, King’s Candy Crush audience make Microsoft the world’s #3 gaming revenue company overnight, and the metaverse language in the announcement is the earlier thesis wearing an acquisition); the distress layer (Activision’s price was discounted by its own crisis, the California harassment litigation and workplace-culture collapse this archive should have filed in 2021 and didn’t, a gap the Fowler thread flags with due shame; Kotick’s exit is priced into the close); and the regulatory layer (the Khan-FTC era gets its defining test case, a trillion-dollar platform buying a content giant, reviewed simultaneously by US, UK, and EU authorities with newly-sharpened doctrine; the file predicts an 18-month gauntlet, behavioral concessions on Call of Duty availability, and ultimate approval, pre-registered, grading in 2023). ...

January 28, 2022

Patch Notes #220 — Year Ten: Green Squares and Golden Mirrors

Year ten of the streak opens with the two best deployment stories imaginable, at opposite scales. A million miles up: JWST’s 344-single-point-of-failure sequence (last year’s held breath) is executing flawlessly, the sunshield (five layers of foil the size of a tennis court, tensioned by remote command) deployed, and this week the primary mirror’s eighteen gold hexagons unfolded and latched. The riskiest zero-rollback deployment in engineering history is, so far, a clean release train; the mirror-alignment phase (months of micro-actuation) begins, and the archive’s professional breath-holding downgrades to professional exhaling. Twenty-five years of rehearsal (that old doctrine at its apex, they tested that sunshield’s every fold in cleanrooms for decades) purchasing fifteen days of flawless production. ...

January 13, 2022

Patch Notes #219 — Year Nine Retrospective: The Plumbing Became the Story

Entry 219 closes year nine, written in the glow of the fortnight’s redemption arc: on Christmas morning, the James Webb Space Telescope launched flawlessly from French Guiana, $10B, 25 years, 344 single-point-of-failure deployment steps now unfolding across a million-mile commute to L2 (the highest-stakes zero-rollback deployment sequence ever attempted; the archive will be following the sunshield tensioning like playoff basketball), carrying the field’s accumulated patience toward the first galaxies. And Spider-Man: No Way Home crossed a billion dollars in a pandemic, proving theatrical mass culture has a pulse when the offering meets the moment (multiverse nostalgia as the portals-scene economy, industrialized). ...

December 29, 2021

Patch Notes #218 — The Log Line That Broke the World

Log4Shell. On December 9th the industry learned that Log4j, the default logging library of the Java ecosystem, embedded in everything from Minecraft servers to Mars-helicopter-adjacent ground systems (yes, really: NASA runs Java too) to every enterprise stack assembled since 2001, would execute code found in log messages: the JNDI lookup feature meant a single crafted string (${jndi:ldap://…}) arriving in any logged field, a username, a User-Agent header, a chat message, a WiFi network name, could pull and run an attacker’s class from an attacker’s server. Remote code execution, via the act of recording what happened. CVSS 10.0. The industry’s collective weekend: cancelled. ...

December 14, 2021

Patch Notes #217 — The Founder Logs Off (Voluntarily, This Time)

Jack Dorsey resigned from Twitter today, his second departure from the company he co-founded, this one voluntary, with a resignation letter arguing against the cult of the founder-CEO itself (“there’s a lot of talk about the importance of a company being ‘founder-led.’ Ultimately I believe that’s severely limiting and a single point of failure,” the man cited SPOF doctrine in his farewell; the archive has never felt more seen by an executive departure). CTO Parag Agrawal inherits the seat, the activist-investor pressure (Elliott’s file has wanted this for years), and the eternal Twitter question: the most culturally load-bearing, financially underperforming platform in the industry (the outrage-optimizer, the deplatforming precedent, the $4.4M-per-word episode, this archive’s index is substantially a Twitter incident log). Dorsey retreats to Block/Square and his bitcoin evangelism (the Web3 wars lose their most interesting combatant on the anti-VC flank). The file’s read: founder-succession is the least-practiced migration in tech (the Basecamp SPOF clause, the incentive-aligned boards), and Twitter is about to run the experiment with maximum observability. The universe, as is its custom with this platform, will make the results interesting beyond anyone’s intent. ...

November 29, 2021

The us-east-1 Problem: Control Planes, Quotas, and a 49-Second CDN Outage

The us-east-1 Problem (Jul 2020 – Sep 2021) Filed November 25, 2021 — the first anniversary of the Kinesis event, the archetypal us-east-1 story. The incidents in this window share a shape. Something small and deep (a thread limit, a quota system, one customer’s config change) spreads out until half the internet feels it. Readers started asking a new first question: what does everything else quietly depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020. Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit. The fleet needed a slow full restart, and services that depended on it (Cognito, CloudWatch, and various vendors’ status pages) went down with it (aws.amazon.com/message/11201). The postmortem taught a lot of engineers what a cell-based architecture is by describing what it looks like to not have one. Google, December 14, 2020. The identity and quota system took down Gmail, YouTube, and Google Cloud auth for about 47 minutes. An automated quota migration reported usage as zero and starved the auth service. Safety systems that can’t tell “no usage” from “no data” became a recurring theme. Slack, January 4, 2021. First workday of the year. Provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021. A Strasbourg datacenter burned, and some customers learned their backups were in the building that was on fire. Physical DR came back into the conversation. Fastly, June 8, 2021. A dormant bug shipped in May was triggered by one customer’s valid config change and dropped about 85% of Fastly’s network. Global outage in seconds, found in minutes, mostly restored within an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together. Forty-nine minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021. A bug triggered by a config update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal Control plane versus data plane became the sharpest way to look at these. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution: in each case the management machinery failed while the actual capacity was fine. “Static stability,” meaning the data plane keeps working when the control plane is down, became the goal to design toward. ...

November 25, 2021 · July 2020 – September 2021 · Retrospective