Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Tue, 01 Jul 2025 10:00:00 +0530

Agents On Call (Jul 2025 – Jul 2026)

This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them.

The incidents defining the period (so far)

AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this.
Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles.
Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit.
Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure.

(This is a living post, updated through July 2026.)

One Regex and a Pandemic: Global Blast Radius Meets Global Load

Mon, 01 Apr 2019 10:00:00 +0530

One Regex and a Pandemic (Apr 2019 – Jun 2020)

This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth.

The incidents that defined the period

Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage.
Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow.
Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream.
Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company.
Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern.
COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test.

What the postmortems reveal

1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews.