Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Tue, 01 Jul 2025 10:00:00 +0530

Agents On Call (Jul 2025 – Jul 2026)

This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them.

The incidents defining the period (so far)

AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this.
Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles.
Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit.
Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure.

(This is a living post, updated through July 2026.)

The us-east-1 Problem: Control Planes, Quotas, and a 49-Second CDN Outage

Wed, 01 Jul 2020 10:00:00 +0530

The us-east-1 Problem (Jul 2020 – Sep 2021)

The incidents of this window share a shape: a small, deep dependency — a thread limit, a quota system, one customer’s config — radiating outward until half the internet notices. Postmortem readers learned to ask a new first question: what does everything else depend on?

The incidents that defined the period

AWS Kinesis / us-east-1, November 25, 2020 — Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit; the fleet needed a slow full restart, and dependent services (Cognito, CloudWatch — and vendors' status pages) failed with it (aws.amazon.com/message/11201). The postmortem taught thousands of engineers what a cell-based architecture is — by describing its absence.
Google, December 14, 2020 — The identity/quota system took down Gmail, YouTube, and Google Cloud auth for ~47 minutes: an automated quota migration reported usage as zero and rationed the auth service to death. Safety systems that can’t distinguish “no usage” from “no data” became a postmortem archetype.
Slack, January 4, 2021 — First workday of the year; provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering).
OVHcloud fire, March 2021 — A Strasbourg datacenter burned; some customers learned their “backups” lived in the building that was on fire. Physical DR returned to the conversation.
Fastly, June 8, 2021 — A dormant bug shipped in May was triggered by one customer’s valid configuration change, dropping ~85% of Fastly’s network. Global outage in seconds; identified in minutes; largely restored in under an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together — 49 minutes that made “CDN concentration” a mainstream news topic.
Akamai Edge DNS, July 2021 — A bug triggered by a configuration update took down banks and airlines for about an hour. Same lesson, different CDN.

What the postmortems reveal

1. Control plane vs data plane became the sharpest lens. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution — in each case the management machinery failed while the underlying capacity was fine. “Static stability” (the data plane keeps working when the control plane is down) became the design goal to cite.