Agents On Call (Jul 2025 – Jul 2026)

This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them.

The incidents defining the period (so far)

  • AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this.
  • Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles.
  • Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit.
  • Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure.

(This is a living post, updated through July 2026.)

What the postmortems reveal

1. The 2017 lessons recurred at 2025 scale. Empty DNS records, oversized config files, edge misconfigurations: none of these failure modes are new. What the autumn proved is that a decade of postmortem culture reduced time-to- transparent-explanation (hours-to-days, with real detail) far more than it reduced occurrence. Complex systems keep manufacturing novel paths to old failures.

2. Automation is now the protagonist. AWS’s DNS race and Cloudflare’s generated feature file were both failures of the machinery that manages the system, not of the system’s capacity. The frontier question in current postmortems: how do humans safely observe, override, and disable automation that is misbehaving? (“Human-on-the-loop” is the term of art.)

3. Concentration risk went political. Governments and regulators responded to the autumn outages with inquiries into cloud dependency for critical services. Expect postmortems increasingly written for statutory audiences (the EU’s DORA regime already requires it in finance).

4. AI-assisted incident response crossed into production. Google’s SRE teams publicly use Gemini-based tooling for incident response and postmortem generation; the vendor ecosystem (incident.io, Rootly, PagerDuty, and a wave of “AI SRE” startups) now ships agents that triage alerts, correlate deploys with error spikes, draft fix PRs, and write the first-draft postmortem. Gartner projects agentic AI operating IT infrastructure at 70% of enterprises by 2029, from under 5% in 2025. The emerging postmortem question — already appearing in 2026 writeups — is “what did the AI responder do, and was it right?”

Practice and tooling shifts

  • AI as first responder, human as reviewer: alert triage, dedup, and enrichment increasingly land on an agent before a human is paged.
  • Postmortem generation automated: timelines assembled from chat, telemetry, and deploy logs; humans edit for judgment and trade-offs rather than reconstructing chronology.
  • Multi-cloud and exit-plan work got funded after the autumn trifecta — less full multi-cloud, more “critical path must survive any one provider.”
  • Hard limits as tested interfaces: Cloudflare’s oversized-file crash is driving teams to treat every hard-coded limit as an input boundary requiring a graceful-degradation test.

Takeaways (interim)

  1. Ask of every automation: how do we know it’s misbehaving, and how do we turn it off without turning off the system?
  2. Anything auto-generated and globally distributed (DNS records, feature files, policies) needs schema validation, size limits with graceful failure, and staged propagation.
  3. If your incident response now includes AI agents, your postmortem template needs a section for their actions — blameless applies to models too, but only if you can reconstruct what they did.
  4. Plan for your provider’s bad month, not just their bad hour: Oct–Nov 2025 put a 14-hour, a 50-hour, and a multi-hour global outage inside five weeks.

Sources for this window