When the Map Burns with the Territory (Oct 2021 – Dec 2022)

Filed January 11, 2023 — finished, fittingly, on the morning the FAA’s NOTAM outage grounded US departures and opened the next chapter.

The image that defines this window: Facebook engineers reportedly couldn’t badge into their own buildings, because the outage had also taken down the door systems. Again and again, the tools meant to fix an outage turned out to depend on the thing that was broken.

The incidents that defined the period

  • Facebook/Meta, October 4, 2021. A routine maintenance command disconnected Facebook’s backbone, and its DNS servers, by design, withdrew their BGP routes when they couldn’t reach the datacenters. Facebook, Instagram, and WhatsApp vanished from the internet for about six hours. Internal tools and remote access died too, so people had to go to the datacenters in person (engineering.fb.com).
  • Roblox, October 28–31, 2021. A 73-hour outage caused by a subtle interaction between a Consul feature and BoltDB performance. The postmortem, co-published with HashiCorp months later, was praised for its depth and for neither party hiding behind the other.
  • AWS us-east-1, December 7, 2021. An automated scaling activity set off a thundering herd on the internal network connecting AWS’s own services, and monitoring and support tooling were among the casualties, which slowed diagnosis (aws.amazon.com/message/12721). Two more December us-east-1 incidents made “why is everything in one region?” a CTO-level question.
  • Log4Shell, December 2021. A logging-library CVE that turned every Java shop’s December into an incident. The response was run like an outage and written up like one, and SBOMs went from an acronym to a mandate.
  • Atlassian, April 2022. A maintenance script given the wrong IDs permanently deleted about 400 customers’ cloud sites. Restoration took up to two weeks because recovery was built for whole-service rollback, not per-customer restore. The postmortem’s honesty about that gap was the lesson.
  • Rogers, July 8, 2022. A maintenance update removed a routing filter, and the resulting BGP flood crashed Canada’s largest network (including 911 and Interac payments) for about a day. National reviews followed, and reliability became a matter of telecom regulation.
  • Cloudflare, June 21, 2022. A BGP change during a datacenter conversion took down 19 of their busiest locations. Postmortem published the same day.
  • UK heatwave, July 2022. Google and Oracle cloud regions in London were throttled by cooling failures. Climate showed up as a reliability factor.
  • Southwest Airlines, December 2022. Crew-scheduling software collapsed under a winter storm, and about 17,000 flights were cancelled. The eventual reckoning, including a record fine, made “legacy system risk” a board agenda item.

What the postmortems reveal

Recovery must not depend on the thing being recovered. Facebook’s DNS, AWS’s monitoring, Atlassian’s restore tooling: each incident dragged on because the repair path ran through the failure. Out-of-band management networks, break-glass access, and offline runbooks became the era’s universal action item.

Deletion needs different engineering than failure. Atlassian could survive crashes fine; it couldn’t un-delete quickly. Soft-delete windows, per-tenant restore, and “recycle bin” behavior for infrastructure entered platform roadmaps.

The supply chain is part of your system. Log4Shell postmortems weren’t really about a bug, they were about inventory: who could answer “where do we run log4j?” in hours instead of weeks.

Public infrastructure expectations arrived. Rogers and Southwest pulled regulators into the postmortem audience, and writeups started being drafted with governments in mind, not just customers.

Practice and tooling shifts

  • Static stability and multi-region moved from AWS whitepaper language to actual funded projects after the December run of incidents.
  • Dependency-mapping tooling surged (service catalogs, Backstage adoption). You can’t reason about a blast radius you can’t see.
  • Learning-from-incidents culture matured, with the Roblox/HashiCorp joint postmortem modeling how a vendor and customer can analyze an incident together.

Takeaways that still hold

  1. Keep a way in that doesn’t depend on your network being up, and test that your badge readers, VPN, and runbooks survive your own outage.
  2. Rate-limit and stage removals (routes, filters, capacity, customers) even more carefully than additions.
  3. Practice per-tenant restore. Whole-system rollback is the wrong grain for multi-tenant platforms.
  4. Keep a dependency inventory you can query during an incident, not after.