October 4th: Facebook, Instagram, and WhatsApp disappeared — not “slow,” not “erroring”: GONE FROM THE INTERNET for nearly six hours, for 3.5 billion users, in the most instructive outage this archive has ever filed (and it files against stiff competition, #102/#206). The mechanism, per Facebook’s own postmortem: a routine backbone-maintenance command (audit tooling bug: the check that should have blocked it, didn’t — #123’s dropdown at backbone scale) disconnected their datacenters from each other; their DNS servers, DESIGNED to withdraw BGP route advertisements when they can’t reach the datacenters (a health-check behaving exactly as specified), dutifully removed Facebook’s nameservers from the internet’s routing table; and then the RECOVERY dependencies detonated in sequence — remote access died with the network it managed, internal tools died with the platform they ran on, and (the detail that made every ops engineer inhale sharply) BADGE READERS on the datacenter doors reportedly failed because they too authenticated against the dead systems. Engineers physically drove to sites and struggled to get into the buildings containing the routers they needed to fix. The map burned with the territory: monitoring, comms, tooling, and DOORS all downstream of the failure they were needed to repair (#175’s own monitor-moving car trip was this lesson at 1/1,000,000th scale). Every org ran the same audit within the week — ours included: our break-glass runbook lived in a wiki whose auth ran through the SSO that a network partition would… yes. It’s printed now. PRINTED. Paper: the ultimate out-of-band.
The outage landed, with novelistic timing, the DAY AFTER Frances Haugen unmasked as the Files’ source (#213) and the DAY BEFORE her Senate testimony — a fortnight in which the company’s systems and its conscience were simultaneously offline. WhatsApp’s six-hour absence also demonstrated concentration risk in its most human denomination: for huge fractions of the Global South, WhatsApp IS the phone system, commerce layer, and family bulletin board — the outage was measured in missed remittances and unreachable parents, not memes (#093’s doctrine, at last count 3.5 billion souls deep).
TIL: BGP withdrawal as designed behavior — the outage’s deepest lesson isn’t a bug; every component did its job. The SYSTEM failed by composition (#140’s chained-bugs doctrine, but for correct behaviors): safety mechanisms that each locally protect can globally amputate. Compose your failsafes on paper, before they compose themselves in production.