Aws

Agents On Call (Jul 2025 – Jul 2026) This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them. The incidents defining the period (so far) AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this. Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles. Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit. Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure. (This is a living post, updated through July 2026.) ...

When the Map Burns with the Territory (Oct 2021 – Dec 2022) The defining image of this window is Facebook engineers reportedly unable to badge into their own buildings because the outage had taken down the systems that controlled the doors. Incident after incident showed recovery tooling, communications, and even physical access welded to the infrastructure they were supposed to repair. The incidents that defined the period Facebook/Meta, October 4, 2021 — A routine maintenance command disconnected Facebook’s backbone; its DNS servers, by design, withdrew their BGP routes when they couldn’t reach the datacenters. Facebook, Instagram, and WhatsApp vanished from the internet for ~6 hours. Internal tools and remote access died too, forcing physical datacenter visits (engineering.fb.com). Roblox, October 28–31, 2021 — A 73-hour outage from a subtle interaction between a Consul feature and BoltDB performance. The postmortem, co-published with HashiCorp months later, was praised for depth and for neither party hiding behind the other. AWS us-east-1, December 7, 2021 — An automated scaling activity triggered a thundering herd on the internal network connecting AWS’s own services; monitoring and support tooling were among the casualties, slowing diagnosis (aws.amazon.com/message/12721). Two further December us-east-1 incidents made “why is everything in one region?” a CTO-level question. Log4Shell, December 2021 — A logging library CVE that turned every Java shop’s December into an incident. The response was run like an outage and postmortem’d like one; SBOMs went from acronym to mandate. Atlassian, April 2022 — A maintenance script given the wrong IDs permanently deleted ~400 customers’ cloud sites; restoration took up to two weeks because recovery was designed for whole-service rollback, not per-customer restore. The postmortem’s candor about that gap was the lesson. Rogers, July 8, 2022 — A maintenance update removed a routing filter and the resulting BGP flood crashed Canada’s largest network — including 911 access and Interac payments — for ~a day. National reviews followed; reliability became telecom regulation. Cloudflare, June 21, 2022 — A BGP change during a datacenter conversion took down 19 of their busiest locations; postmortem published same day. UK heatwave, July 2022 — Google and Oracle cloud regions in London throttled by cooling failures: climate as a reliability factor. Southwest Airlines, December 2022 — Crew-scheduling software collapsed under a winter storm; ~17,000 flights cancelled. The eventual reckoning (including a record fine) made “legacy system risk” a board agenda item. What the postmortems reveal 1. Recovery must not depend on the thing being recovered. Facebook’s DNS, AWS’s monitoring, Atlassian’s restore tooling — each incident extended because the repair path ran through the failure. Out-of-band management networks, break-glass access, and offline runbooks became the era’s universal action item. ...

The us-east-1 Problem (Jul 2020 – Sep 2021) The incidents of this window share a shape: a small, deep dependency — a thread limit, a quota system, one customer’s config — radiating outward until half the internet notices. Postmortem readers learned to ask a new first question: what does everything else depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020 — Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit; the fleet needed a slow full restart, and dependent services (Cognito, CloudWatch — and vendors' status pages) failed with it (aws.amazon.com/message/11201). The postmortem taught thousands of engineers what a cell-based architecture is — by describing its absence. Google, December 14, 2020 — The identity/quota system took down Gmail, YouTube, and Google Cloud auth for ~47 minutes: an automated quota migration reported usage as zero and rationed the auth service to death. Safety systems that can’t distinguish “no usage” from “no data” became a postmortem archetype. Slack, January 4, 2021 — First workday of the year; provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021 — A Strasbourg datacenter burned; some customers learned their “backups” lived in the building that was on fire. Physical DR returned to the conversation. Fastly, June 8, 2021 — A dormant bug shipped in May was triggered by one customer’s valid configuration change, dropping ~85% of Fastly’s network. Global outage in seconds; identified in minutes; largely restored in under an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together — 49 minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021 — A bug triggered by a configuration update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal 1. Control plane vs data plane became the sharpest lens. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution — in each case the management machinery failed while the underlying capacity was fine. “Static stability” (the data plane keeps working when the control plane is down) became the design goal to cite. ...

Typos That Broke the Internet (Oct 2016 – Dec 2017) If one window proved that public, honest postmortems build more trust than they cost, it’s this one. A livestreamed database recovery and a typo that took down half the web produced two of the most-read incident reports in history. The incidents that defined the period Dyn DNS DDoS, October 21, 2016 — The Mirai botnet, built from IoT devices, took down a major managed-DNS provider and with it Twitter, Netflix, Reddit, and GitHub for much of a day. The industry’s introduction to dependency concentration: dozens of “independent” sites shared one DNS provider. GitLab database incident, January 31, 2017 — An exhausted engineer, fighting replication lag, ran rm -rf on the primary’s data directory. Five backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute postmortem (about.gitlab.com). ~6 hours of data was lost — and GitLab’s reputation arguably improved. AWS S3 us-east-1, February 28, 2017 — An operator debugging the billing system mistyped a playbook parameter and removed far more capacity than intended; the index subsystem required a full restart it hadn’t had in years (aws.amazon.com/message/41926). Thousands of sites broke — including, memorably, AWS’s own status page, whose health icons were hosted on S3. Cloudbleed, February 2017 — A parser bug leaked memory across Cloudflare customers into cached pages. Cloudflare’s forensic-grade disclosure set a new bar for security postmortems. British Airways, May 2017 — A datacenter power event (a contractor and a UPS) grounded flights globally; the vague public explanation became the counterexample to GitLab-style transparency. Equifax breach, 2017 — An unpatched Struts vulnerability; the postmortem lesson was less about the bug than about asset inventory and patch governance. What the postmortems reveal 1. Transparency won, decisively. GitLab and AWS gave specifics (the command, the parameter, the safety checks now added); BA gave vagueness. The market noticed which companies it trusted more afterward. “Publish the real postmortem” became a competitive strategy, not a legal risk. ...