Agents On Call (Jul 2025 – Aug 2026)

This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them.

The incidents defining the period (so far)

  • Cloudflare 1.1.1.1, July 14, 2025. The window’s opening act, and the cleanest example of its central pattern. A June 6 change preparing a new Data Localization Suite service accidentally listed the 1.1.1.1 resolver’s IP prefixes. The error sat dormant for five weeks. On July 14, adding an offline test location triggered a global config refresh, which withdrew every 1.1.1.1 prefix from Cloudflare’s network at once. 62 minutes of global DNS failure (21:52–22:54 UTC), on one of the most-used resolvers on the internet. Cloudflare’s postmortem was public within a day, and had to spend a section explaining that a real BGP hijack of the prefix by another AS during the outage was a coincidence, not the cause.
  • AWS us-east-1, October 20, 2025. A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint. The automation couldn’t fix itself, and the failure cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption, and Snapchat alone drew about 3 million outage reports. The biggest us-east-1 event since December 2021, and a case study in automation deadlock: the fix meant humans disabling the automation that was supposed to prevent exactly this.
  • Azure Front Door, October 29, 2025. An accidental configuration change broke Microsoft’s global edge and CDN layer for about eight hours, taking down the Azure portal, M365 entry points, and customer sites, days before earnings and a week after AWS’s turn. A separate East US2 networking config outage lasting about 50 hours showed that regional incidents can now outlast the news cycle.
  • Cloudflare, November 18, 2025. A database permissions change caused the Bot Management feature file to double in size and blow past a hard-coded limit in the core proxy, and processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally generated file, pushed globally, hitting an untested limit.
  • Cloudflare, December 5, 2025 and February 20, 2026. A roughly 25-minute traffic outage, then a BGP withdrawal affecting bring-your-own-IP customers. Smaller events, but notable for how fast and detailed the disclosures now are.

July 2026: four providers, nine days

  • AWS CloudFront, July 16, 2026. An internal constraint on the fleet managing connections to private VPC origins meant updated network configuration failed to load into the packet processors. VPC Origins — the 2024 feature that lets CloudFront pull from private subnets — broke widely, and because CloudFront is a CDN, the blast radius was other people’s SaaS: Hugging Face, Canvas, Blackboard, Frontegg, Coda, Ubiquiti. 3 hours 33 minutes (07:45–11:18 UTC), with 96 minutes between the first 5xx reports and AWS isolating VPC Origins as the culprit. The in-incident workaround was to stop using the feature.
  • Google Cloud europe-west4-a, July 20, 2026. The window’s first purely physical failure: a power disturbance took out backup power and cooling, and equipment shut itself down to avoid heat damage. 14 hours 55 minutes, hitting 24 VMware Engine private clouds, 9 Bare Metal Solution customers, and 6 NetApp Volumes clusters. A separate July 14 VMware Engine incident had already caused BGP instability and witness-appliance loss via a network config update.
  • Azure West US, July 23, 2026. A maintenance process removed more network routes than intended — connectivity failures and elevated latency from 14:44 to 19:41 UTC. Workloads already inside the region kept running; reaching them was the problem. The Facebook 2021 lesson, at five hours and one region.
  • AWS us-west-2, July 24, 2026. A regional internet-connectivity failure in Oregon took Apple Pay, Reddit, Hulu, DoorDash, and PlayStation Network offline for about 80 minutes. Short, but it was AWS’s third distinct reliability incident in eleven weeks, which is the part the trade press led with.

(This is a living post, updated through August 6, 2026.)

What the postmortems reveal

The 2017 lessons came back at 2025 scale. Empty DNS records, oversized config files, edge misconfigurations: none of these failure modes are new. What the autumn proved is that a decade of postmortem culture cut the time to a clear, honest explanation (now hours to days, with real detail) far more than it cut how often these things happen. Complex systems keep finding new routes to old failures.

Automation is now the main character. AWS’s DNS race and Cloudflare’s generated feature file were both failures of the machinery that manages the system, not of the system’s capacity. July 2026 extended the pattern to routine maintenance: Azure’s West US outage was a maintenance process withdrawing more routes than it meant to, and CloudFront’s was a configuration distribution that failed to load. The live question in current postmortems is how humans can safely watch, override, and shut off automation that’s misbehaving. The term of art is “human-on-the-loop.”

Latency to detection is the metric that hasn’t improved. The 1.1.1.1 error waited five weeks for a trigger. CloudFront took 96 minutes to isolate a feature that only some customers used. The industry got very good at explaining incidents afterward and has not gotten meaningfully better at noticing them sooner, because the telemetry is aimed at capacity and these failures are in the control plane.

Not everything is software. The single longest outage of July 2026 was a power disturbance in a Google Cloud zone that defeated backup power and cooling, and the machines shut themselves down to avoid cooking. Fifteen hours, no bad config involved. As buildout accelerates against strained grids, the physical-layer postmortem is worth keeping in the rotation alongside the config-plane one.

Concentration risk went political. Governments and regulators responded to the autumn outages with inquiries into how much critical infrastructure depends on a few clouds. Expect more postmortems written for legal audiences, since the EU’s DORA regime already requires it in finance.

AI-assisted incident response crossed into production. Google’s SRE teams publicly use Gemini-based tooling for incident response and postmortem writing, and the vendor ecosystem (incident.io, Rootly, PagerDuty, and a wave of “AI SRE” startups) now ships agents that triage alerts, connect deploys to error spikes, draft fix PRs, and write the first-draft postmortem. Gartner projects agentic AI running IT infrastructure at 70% of enterprises by 2029, up from under 5% in 2025. The new postmortem question, already showing up in 2026 writeups, is “what did the AI responder do, and was it right?”

Practice and tooling shifts

  • AI as first responder, human as reviewer. Alert triage, dedup, and enrichment increasingly happen on an agent before a human is paged.
  • Postmortem generation is getting automated. Timelines are assembled from chat, telemetry, and deploy logs, and humans edit for judgment and trade-offs instead of reconstructing the chronology by hand.
  • Multi-cloud and exit-plan work got funded after the autumn trifecta. Less full multi-cloud, more “the critical path must survive any one provider.”
  • Hard limits are being treated as tested interfaces. Cloudflare’s oversized-file crash is pushing teams to treat every hard-coded limit as an input boundary that needs a graceful-degradation test.

Takeaways (interim)

  1. For every automation, ask how you’ll know it’s misbehaving and how you’ll turn it off without turning off the system.
  2. Anything auto-generated and globally distributed (DNS records, feature files, policies) needs schema validation, size limits that fail gracefully, and staged propagation.
  3. If your incident response now includes AI agents, your postmortem template needs a section for what they did. Blameless applies to models too, but only if you can reconstruct their actions.
  4. Plan for your provider’s bad month, not just their bad hour. October and November 2025 packed a 14-hour, a 50-hour, and a multi-hour global outage into five weeks; July 2026 put four providers on the board in nine days.
  5. Know which of your provider’s features you depend on, not just which provider. CloudFront didn’t fail — VPC Origins did, and only customers using it noticed until the downstream SaaS they relied on went with it. Dependency inventories that stop at the vendor name are half an inventory.
  6. Ask your cloud where the power comes from. europe-west4-a was the window’s longest single outage and had nothing to do with code.

Sources for this window