When the Map Burns with the Territory: BGP Lockouts and Cascading Dependencies

Fri, 01 Oct 2021 10:00:00 +0530

When the Map Burns with the Territory (Oct 2021 – Dec 2022)

The defining image of this window is Facebook engineers reportedly unable to badge into their own buildings because the outage had taken down the systems that controlled the doors. Incident after incident showed recovery tooling, communications, and even physical access welded to the infrastructure they were supposed to repair.

The incidents that defined the period

Facebook/Meta, October 4, 2021 — A routine maintenance command disconnected Facebook’s backbone; its DNS servers, by design, withdrew their BGP routes when they couldn’t reach the datacenters. Facebook, Instagram, and WhatsApp vanished from the internet for ~6 hours. Internal tools and remote access died too, forcing physical datacenter visits (engineering.fb.com).
Roblox, October 28–31, 2021 — A 73-hour outage from a subtle interaction between a Consul feature and BoltDB performance. The postmortem, co-published with HashiCorp months later, was praised for depth and for neither party hiding behind the other.
AWS us-east-1, December 7, 2021 — An automated scaling activity triggered a thundering herd on the internal network connecting AWS’s own services; monitoring and support tooling were among the casualties, slowing diagnosis (aws.amazon.com/message/12721). Two further December us-east-1 incidents made “why is everything in one region?” a CTO-level question.
Log4Shell, December 2021 — A logging library CVE that turned every Java shop’s December into an incident. The response was run like an outage and postmortem’d like one; SBOMs went from acronym to mandate.
Atlassian, April 2022 — A maintenance script given the wrong IDs permanently deleted ~400 customers’ cloud sites; restoration took up to two weeks because recovery was designed for whole-service rollback, not per-customer restore. The postmortem’s candor about that gap was the lesson.
Rogers, July 8, 2022 — A maintenance update removed a routing filter and the resulting BGP flood crashed Canada’s largest network — including 911 access and Interac payments — for ~a day. National reviews followed; reliability became telecom regulation.
Cloudflare, June 21, 2022 — A BGP change during a datacenter conversion took down 19 of their busiest locations; postmortem published same day.
UK heatwave, July 2022 — Google and Oracle cloud regions in London throttled by cooling failures: climate as a reliability factor.
Southwest Airlines, December 2022 — Crew-scheduling software collapsed under a winter storm; ~17,000 flights cancelled. The eventual reckoning (including a record fine) made “legacy system risk” a board agenda item.

What the postmortems reveal

1. Recovery must not depend on the thing being recovered. Facebook’s DNS, AWS’s monitoring, Atlassian’s restore tooling — each incident extended because the repair path ran through the failure. Out-of-band management networks, break-glass access, and offline runbooks became the era’s universal action item.

One Regex and a Pandemic: Global Blast Radius Meets Global Load

Mon, 01 Apr 2019 10:00:00 +0530

One Regex and a Pandemic (Apr 2019 – Jun 2020)

This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth.

The incidents that defined the period

Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage.
Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow.
Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream.
Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company.
Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern.
COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test.

What the postmortems reveal

1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews.

Bgp on Azarudeen.com

When the Map Burns with the Territory: BGP Lockouts and Cascading Dependencies

When the Map Burns with the Territory (Oct 2021 – Dec 2022)

The incidents that defined the period

What the postmortems reveal

One Regex and a Pandemic: Global Blast Radius Meets Global Load

One Regex and a Pandemic (Apr 2019 – Jun 2020)

The incidents that defined the period

What the postmortems reveal