In late August the Milwaukee Bucks refused to take the floor for a playoff game in protest of racial injustice, and within hours the entire NBA postseason had stopped, followed by baseball, soccer, and tennis — Naomi Osaka declaring she wouldn’t play her Western & Southern semifinal, which forced the tournament to pause with her. Athletes withholding the product, on live TV, across four sports in a day. Play resumed days later with concrete commitments negotiated by the players. The archive’s values-stack-trace methodology grades this one cleanly: the value executed at the product layer, the product being the games themselves, withheld. Whatever one’s politics, the staff-engineering observation stands on its own: leverage is positional. The closed environments that isolate players also concentrate them, one tour, every star, cameras pre-installed, turning a labor action that would once have required season-long coordination into a unanimous decision. Constraints reshape power; whoever designs the system designs the strike conditions too, usually without knowing it. ...

Epic pulled the trigger on the war it’s been loading for a while: on August 13th, Fortnite shipped a direct-payment option in violation of App Store rules; Apple removed Fortnite within hours; and Epic, revealing the entire operation was rehearsed (the doctrine weaponized), instantly filed a prepared antitrust lawsuit and premiered a shot-for-shot parody of Apple’s own iconic “1984” ad, recast with Apple as Big Brother, in Fortnite, to an audience of millions of teenagers now chanting “Free Fortnite.” Google removed it too and received its own pre-drafted lawsuit as a party favor. As choreographed corporate combat, it’s the best-produced opening move this archive has ever filed: Epic converted a contract violation into a cultural event with discovery documents attached, betting that the antitrust weather turns courts and Congress into tailwinds. Apple’s 30% now faces its most dangerous opponent: not a regulator, but a plaintiff with subpoena power, a war chest, and a propaganda channel installed on every teenager’s device. Multi-year case; the file is opened and labeled load-bearing. ...

The antitrust hearing happened: Bezos, Zuckerberg, Cook, and Pichai, four trillion-ish dollars of market cap testifying by video call (the format itself a pandemic artifact: history’s most powerful men glitching on mute like the rest of us) before a House subcommittee that had done, for once, actual homework: internal emails on the Instagram acquisition (“neutralize a competitor,” in writing, the email-is-forever lesson ascending to antitrust exhibit), Amazon’s use of third-party seller data against its own sellers, App Store tax mechanics (the Fortnite war, now congressional), and Google’s search self-preferencing. The staff-level observation: this hearing differed from 2018’s (“Senator, we run ads”) in preparation asymmetry, the questioners had documents, timelines, and follow-ups, and the survived-theater playbook visibly strained against subpoenaed receipts. The report and the suits come later (they always come later; the fog of task forces), but the frame has shifted: the question on screen was no longer “what is your business model?” but “should you be allowed to have all of it?” ...

On July 15th, the Twitter accounts of Barack Obama, Joe Biden, Elon Musk, Bill Gates, Apple, and Uber all tweeted the same Bitcoin scam within an hour, and the mechanism, now confirmed, was not exotic: attackers social-engineered Twitter employees (phone spear-phishing) into internal admin tools, and those tools could post as any account. The haul was a laughable ~$120k in BTC; the demonstrated capability, tweet as a president-elect, as a head of state, as the SEC’s favorite CEO, during a market day or a crisis, was priceless, which is exactly the asymmetry that should terrify everyone (the attackers were, mercifully and per early reporting, after pocket money rather than a war pretext). The staff-file lessons, all archive reruns at maximum blast radius: internal tools are production (the admin panel is the most privileged deploy surface you own and the least reviewed); support-staff access is the perimeter (the inside-the-VPC call); and “god mode” tooling needs the same guardrails as capacity removal, scoping, approvals, session recording, and alarms on volume. We audited our own admin panel Thursday. Findings: three actions with no audit log. Fixed by Monday. The industry’s collective admin-panel audit this week may be the hack’s real payload. ...

The humming thread is no longer humming; it’s demoing. OpenAI’s GPT-3 API went into beta last month and my Twitter feed has spent the fortnight full of demos that would have been dismissed as staged a year ago: plain-English descriptions turned into working UI code; legal prose translated to plain language; essays continued in the style of their opening paragraph; the beta invite queue is currently the hottest ticket in tech. It’s “just” next-token prediction at 175 billion parameters, and the honest staff-level assessment after two weeks of playing with it: it is simultaneously the most impressive parlor trick I’ve ever used and obviously, obviously, a new kind of infrastructure. It confabulates with total confidence (ask it for citations and it invents plausible ones, the failure mode is fluency, which is worse than being wrong badly). It has no idea what it doesn’t know. And none of that changes the trajectory: the Transformer paper, the BERT benchmark, the staged release, and this entry logs the first fortnight I watched non-ML engineers build products on the thread. The extrapolation exercise is now everyone’s homework. ...

One Regex and a Pandemic (Apr 2019 – Jun 2020) Filed July 2, 2020 — one year after the 27-minute regex outage this window is named for. This window has a clean shape. It opens with self-inflicted global outages at Cloudflare and Google that pushed the industry toward staged rollouts, and it closes with COVID-19 stress-testing every capacity plan on the planet. The incidents that defined the period Google Cloud, June 2, 2019. A maintenance automation event descheduled network control-plane jobs across several regions, and the resulting congestion throttled Google Cloud, YouTube, and Gmail for about four hours. The detail everyone remembers: the outage broke the very tools engineers needed to fix it. Cloudflare, July 2, 2019. A single WAF rule with a badly backtracking regex went out globally (WAF rules skipped staged rollout on purpose, for fast response to attacks) and pinned every CPU on Cloudflare’s edge. Twenty-seven minutes of global 502s, and one of the best postmortems ever written (blog.cloudflare.com), including a short lecture on regex complexity and why the kill switch was slow. Verizon BGP route leak, June 24, 2019. A small ISP’s route optimizer leaked routes through Verizon and blackholed chunks of the internet, Cloudflare included. Cloudflare’s blunt writeup (it called the incident “a small heart attack”) pushed RPKI adoption into the mainstream. Stripe, July 2019. Two linked database failures. Stripe published a detailed root-cause report, which was notable for a payments company. Salesforce, May 2019. A database script granted broad permissions across orgs, and the fix (revoking those permissions widely) caused more disruption than the bug. Recovery-as-a-second-incident got a name. COVID-19 surge, March–June 2020. Zoom grew about 30x. Robinhood had repeated trading-day outages from launch-day architecture under thundering-herd load. Streaming services in Europe voluntarily dropped quality, and unemployment systems on mainframes buckled. It was less one incident than a global load test. What the postmortems reveal Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose, for speed against attackers. The lesson wasn’t “never ship fast,” it was “your fastest pipeline needs the strongest safety switches.” Anything that goes global instantly became a flag in design reviews. ...

WWDC today (virtual, prerecorded, and honestly sharper for it, the pandemic’s conference format may outlive the pandemic): Apple announced the Mac is leaving Intel for Apple Silicon, their own ARM chips, the iPhone’s architecture ascending to the desktop, a two-year transition with Rosetta 2 translation for the legacy world. This is the third architecture migration in Mac history (68k to PowerPC to Intel to ARM) and the old deprecation doctrine applies at maximum scale: they announced it with tooling ready (universal binaries, translation layer, dev transition kits shipping now), the Adobe-Flash three-year-hospice model, executed by the company that owns the whole stack. My prediction, filed for grading: the performance-per-watt numbers from the iPhone lineage suggest the first ARM Macs won’t be “acceptable substitutes” but flatly faster than the Intel ones they replace, which will make this the rare migration users request rather than tolerate. (The Apple-$2T-by-2022 sweepstake entry is feeling comfortable.) ...

Two Americas shared a split screen this fortnight, and the archive files both because both are true. Wednesday’s attempt scrubbed at T-minus-17 minutes on weather, exactly as the last entry hoped it would if the data said so. Then Saturday May 30th: Crew Dragon flew, Bob and Doug to orbit in a capsule with touchscreens, the Falcon 9 booster landing downrange on schedule like the miracle had become a bus route (the crying kid now watches landings with a checklist), dock, hatch open, nine years of gap closed by a company that iterated through explosions in public. The same week: George Floyd was killed by Minneapolis police, and the country erupted into the largest protest movement in its modern history, ongoing as I write, in every city including ours. ...

The fortnight ahead holds the first attempt to launch American astronauts from American soil since the Shuttle retired in 2011, SpaceX’s Crew Dragon Demo-2, Bob and Doug (the actual names; the universe casts well), scheduled Wednesday from pad 39A itself. Nine years of gap, bridged by the company this archive watched land its first booster through a haze of my own tears. Florida’s weather forecast is hostile all week, so I’m pre-registering the healthy outcome: if they scrub with humans aboard, that’s the system working (the inverse of the certification-shortcut file; a launch culture that treats schedule pressure as a first-class hazard; “the rocket doesn’t care about your press availability” is the healthiest sentence in aerospace). The whole locked-down planet could use this launch, which is precisely the pressure the scrub discipline exists to ignore. ...

The layoff wave arrived. Travel and mobility first, as their revenue approaches literal zero: Airbnb cut ~25% this week, Uber ~14%, with more queued behind them. The archive files layoffs under the old asymmetric-loyalty lesson, but this fortnight adds a craft observation, because the variance in execution was enormous and instructive. Airbnb’s version, a long, specific, personally-signed letter explaining the reasoning, generous severance, healthcare extensions, an alumni talent directory the company built and staffed recruiters to, actively placing its own laid-off people, was immediately and correctly held up as the standard. The cynical read (“great PR”) and the sincere read (“great leadership”) converge on the same operational fact: how you execute your worst day is a design decision you make before the worst day, with runbooks, like everything else this blog believes in (the purchasing postmortem has a sibling: every layoff postmortem is a values postmortem, the runtime-configuration lesson at maximum stakes). I took notes I hope never to use. ...