Facebook

When the Map Burns with the Territory (Oct 2021 – Dec 2022) The defining image of this window is Facebook engineers reportedly unable to badge into their own buildings because the outage had taken down the systems that controlled the doors. Incident after incident showed recovery tooling, communications, and even physical access welded to the infrastructure they were supposed to repair. The incidents that defined the period Facebook/Meta, October 4, 2021 — A routine maintenance command disconnected Facebook’s backbone; its DNS servers, by design, withdrew their BGP routes when they couldn’t reach the datacenters. Facebook, Instagram, and WhatsApp vanished from the internet for ~6 hours. Internal tools and remote access died too, forcing physical datacenter visits (engineering.fb.com). Roblox, October 28–31, 2021 — A 73-hour outage from a subtle interaction between a Consul feature and BoltDB performance. The postmortem, co-published with HashiCorp months later, was praised for depth and for neither party hiding behind the other. AWS us-east-1, December 7, 2021 — An automated scaling activity triggered a thundering herd on the internal network connecting AWS’s own services; monitoring and support tooling were among the casualties, slowing diagnosis (aws.amazon.com/message/12721). Two further December us-east-1 incidents made “why is everything in one region?” a CTO-level question. Log4Shell, December 2021 — A logging library CVE that turned every Java shop’s December into an incident. The response was run like an outage and postmortem’d like one; SBOMs went from acronym to mandate. Atlassian, April 2022 — A maintenance script given the wrong IDs permanently deleted ~400 customers’ cloud sites; restoration took up to two weeks because recovery was designed for whole-service rollback, not per-customer restore. The postmortem’s candor about that gap was the lesson. Rogers, July 8, 2022 — A maintenance update removed a routing filter and the resulting BGP flood crashed Canada’s largest network — including 911 access and Interac payments — for ~a day. National reviews followed; reliability became telecom regulation. Cloudflare, June 21, 2022 — A BGP change during a datacenter conversion took down 19 of their busiest locations; postmortem published same day. UK heatwave, July 2022 — Google and Oracle cloud regions in London throttled by cooling failures: climate as a reliability factor. Southwest Airlines, December 2022 — Crew-scheduling software collapsed under a winter storm; ~17,000 flights cancelled. The eventual reckoning (including a record fine) made “legacy system risk” a board agenda item. What the postmortems reveal 1. Recovery must not depend on the thing being recovered. Facebook’s DNS, AWS’s monitoring, Atlassian’s restore tooling — each incident extended because the repair path ran through the failure. Out-of-band management networks, break-glass access, and offline runbooks became the era’s universal action item. ...

When Automation Fights Back (Jan 2018 – Mar 2019) By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from “why did the component fail?” to “why did our self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018 — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain. GitHub, October 21, 2018 — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. Split-brain. GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (github.blog). Microsoft Azure South Central US, September 2018 — A lightning strike caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many “global” Azure services (including Azure AD and the status portal) had hidden dependencies on one region. Google Cloud, July 2018 — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius. Facebook, March 13, 2019 — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a server configuration change — at the time the longest outage in the company’s history. Wells Fargo, February 2019 — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal 1. Automated failover needs a theory of data. GitHub’s incident became the case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking “what does our orchestrator do during a partition?” — a Jepsen-style question applied to ops tooling. ...