Agents On Call (Jul 2025 – Sep 2026)

Filed September 7, 2026. This window is closed; October 2026 onward continues in part 12.

This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Then August and early September added two things this window had not seen: outages caused by plain demand rather than bad configuration, and the first big test of the multi-provider hedge everyone adopted after the autumn — which it failed. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them.

The incidents defining the period

  • Cloudflare 1.1.1.1, July 14, 2025. The window’s opening act, and the cleanest example of its central pattern. A June 6 change preparing a new Data Localization Suite service accidentally listed the 1.1.1.1 resolver’s IP prefixes. The error sat dormant for five weeks. On July 14, adding an offline test location triggered a global config refresh, which withdrew every 1.1.1.1 prefix from Cloudflare’s network at once. 62 minutes of global DNS failure (21:52–22:54 UTC), on one of the most-used resolvers on the internet. Cloudflare’s postmortem was public within a day, and had to spend a section explaining that a real BGP hijack of the prefix by another AS during the outage was a coincidence, not the cause.
  • AWS us-east-1, October 20, 2025. A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint. The automation couldn’t fix itself, and the failure cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption, and Snapchat alone drew about 3 million outage reports. The biggest us-east-1 event since December 2021, and a case study in automation deadlock: the fix meant humans disabling the automation that was supposed to prevent exactly this.
  • Azure Front Door, October 29, 2025. An accidental configuration change broke Microsoft’s global edge and CDN layer for about eight hours, taking down the Azure portal, M365 entry points, and customer sites, days before earnings and a week after AWS’s turn. A separate East US2 networking config outage lasting about 50 hours showed that regional incidents can now outlast the news cycle.
  • Cloudflare, November 18, 2025. A database permissions change caused the Bot Management feature file to double in size and blow past a hard-coded limit in the core proxy, and processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally generated file, pushed globally, hitting an untested limit.
  • Cloudflare, December 5, 2025 and February 20, 2026. A roughly 25-minute traffic outage, then a BGP withdrawal affecting bring-your-own-IP customers. Smaller events, but notable for how fast and detailed the disclosures now are.

July 2026: four providers, nine days

  • AWS CloudFront, July 16, 2026. An internal constraint on the fleet managing connections to private VPC origins meant updated network configuration failed to load into the packet processors. VPC Origins — the 2024 feature that lets CloudFront pull from private subnets — broke widely, and because CloudFront is a CDN, the blast radius was other people’s SaaS: Hugging Face, Canvas, Blackboard, Frontegg, Coda, Ubiquiti. 3 hours 33 minutes (07:45–11:18 UTC), with 96 minutes between the first 5xx reports and AWS isolating VPC Origins as the culprit. The in-incident workaround was to stop using the feature.
  • Google Cloud europe-west4-a, July 20, 2026. The window’s first purely physical failure: a power disturbance took out backup power and cooling, and equipment shut itself down to avoid heat damage. 14 hours 55 minutes, hitting 24 VMware Engine private clouds, 9 Bare Metal Solution customers, and 6 NetApp Volumes clusters. A separate July 14 VMware Engine incident had already caused BGP instability and witness-appliance loss via a network config update.
  • Azure West US, July 23, 2026. A maintenance process removed more network routes than intended — connectivity failures and elevated latency from 14:44 to 19:41 UTC. Workloads already inside the region kept running; reaching them was the problem. The Facebook 2021 lesson, at five hours and one region.
  • AWS us-west-2, July 24, 2026. A regional internet-connectivity failure in Oregon took Apple Pay, Reddit, Hulu, DoorDash, and PlayStation Network offline for about 80 minutes. Short, but it was AWS’s third distinct reliability incident in eleven weeks, which is the part the trade press led with.

August–September 2026: capacity, and a correlated failure

  • GitHub, August 6 and August 17, 2026. Two outages in one month, and neither involved a code or configuration change. The August 17 event ran 7 hours 47 minutes across github.com, authentication, Actions, the API, and Copilot, with web and API error rates peaking near 20% and raw/archive downloads near 50%. Record traffic pushed the Istio sidecar proxies that mediate service-to-service calls to their processing ceiling, and autoscaling did not trigger because it did not account for sidecar capacity. The shortfall cascaded into HAProxy and then authentication. During recovery, a VS Code retry bug turned waiting clients into a load generator and extended the outage. GitHub’s writeup commits to adding 3 million CPU cores and 120 petabytes of storage, auditing Istio limits, fixing retry and backoff behavior, and isolating critical systems.
  • Google Cloud us-west1, August 20, 2026. Scheduled fiber maintenance unexpectedly reduced network capacity between data centers in the region, and the automated rerouting that exists to redistribute traffic during maintenance did not do so correctly. Congestion turned into throttling and latency spikes across 33 products, including Compute Engine, GKE, Cloud SQL, BigQuery, Cloud Storage, and IAM. Google logged 2 hours 22 minutes; third-party monitors saw closer to 3 hours 40. The incident report’s key admission is that the automated systems built to catch this class of fault did not catch it.
  • Google Cloud us-central1-b, September 1, 2026. A single-zone network service degradation reaching 15 products over 4 hours 8 minutes — a reminder that “one zone” is not a small blast radius when customers treat zones as interchangeable.
  • ChatGPT, Claude, and Grok, September 3, 2026. Three competing AI providers degraded inside the same window of roughly 90 minutes to two hours, taking Codex, Cursor, and other agent tooling with them; Copilot reported problems and Gemini was largely unaffected. Microsoft reported a network infrastructure incident in Azure’s East US region during the same period, and multiple outlets attributed the AI outages to it, but at the close of this window no provider had confirmed a shared cause and no postmortem had been published. Anthropic had also had a separate 42-minute authentication outage on August 17.

What the postmortems reveal

The 2017 lessons came back at 2025 scale. Empty DNS records, oversized config files, edge misconfigurations: none of these failure modes are new. What the autumn proved is that a decade of postmortem culture cut the time to a clear, honest explanation (now hours to days, with real detail) far more than it cut how often these things happen. Complex systems keep finding new routes to old failures.

Capacity came back. For a year this window’s story was that outages had moved into the control plane: config artifacts, DNS automation, policy distribution. August broke that pattern. GitHub went down twice in a month from plain demand, with no bad deploy involved, and named the growth of AI-assisted development as part of why traffic reached records. That is worth sitting with. When a large share of your clients are agents rather than people, the load curve stops looking like the one your capacity model was fitted on: it has no evenings, no weekends, and it retries far more patiently than a human would. Capacity planning was supposed to be the solved, boring part.

Safety automation is failing to engage, which is a different problem from automation causing outages. GitHub’s autoscaler did not fire because it scaled on services and the ceiling was in the sidecars mediating them. Google’s automated rerouting during scheduled maintenance did not redistribute traffic correctly, and the fault detection built for exactly that class of failure did not catch it. In both cases the protective mechanism was present, configured, and irrelevant, because it was watching a dimension that was not the one that saturated.

Recovery now depends on client behavior. GitHub’s outage was extended by a VS Code retry bug that turned every waiting editor into a load generator the moment service began returning. Your clients’ backoff logic is part of your recovery time, and for most platforms that logic ships inside somebody else’s IDE, SDK, or agent framework.

Automation is now the main character. AWS’s DNS race and Cloudflare’s generated feature file were both failures of the machinery that manages the system, not of the system’s capacity. July 2026 extended the pattern to routine maintenance: Azure’s West US outage was a maintenance process withdrawing more routes than it meant to, and CloudFront’s was a configuration distribution that failed to load. The live question in current postmortems is how humans can safely watch, override, and shut off automation that’s misbehaving. The term of art is “human-on-the-loop.”

Latency to detection is the metric that hasn’t improved. The 1.1.1.1 error waited five weeks for a trigger. CloudFront took 96 minutes to isolate a feature that only some customers used. The industry got very good at explaining incidents afterward and has not gotten meaningfully better at noticing them sooner, because the telemetry watches the serving path while these failures live in the control plane — or, as August showed, in a capacity dimension nobody thought to graph.

Not everything is software. The single longest outage of July 2026 was a power disturbance in a Google Cloud zone that defeated backup power and cooling, and the machines shut themselves down to avoid cooking. Fifteen hours, no bad config involved. As buildout accelerates against strained grids, the physical-layer postmortem is worth keeping in the rotation alongside the config-plane one.

Correlated failure arrived at the model layer. The standard response to 2025’s outage autumn was to spread risk across providers, and for AI specifically that meant running two or three labs behind a routing layer with evals to keep quality honest. September 3 was the first real test of that hedge: ChatGPT, Claude, and Grok degraded inside the same window. No shared cause has been confirmed, and it may yet turn out to be coincidence. But the strategy assumes the providers fail independently, almost nobody has evidence that they do, and there is currently no practical way for a customer to obtain that evidence before an incident supplies it. A portfolio of vendors sitting on the same underlying infrastructure is one vendor with three invoices.

Concentration risk went political. Governments and regulators responded to the autumn outages with inquiries into how much critical infrastructure depends on a few clouds. Expect more postmortems written for legal audiences, since the EU’s DORA regime already requires it in finance.

AI-assisted incident response crossed into production. Google’s SRE teams publicly use Gemini-based tooling for incident response and postmortem writing, and the vendor ecosystem (incident.io, Rootly, PagerDuty, and a wave of “AI SRE” startups) now ships agents that triage alerts, connect deploys to error spikes, draft fix PRs, and write the first-draft postmortem. Gartner projects agentic AI running IT infrastructure at 70% of enterprises by 2029, up from under 5% in 2025. The new postmortem question, already showing up in 2026 writeups, is “what did the AI responder do, and was it right?”

Practice and tooling shifts

  • Autoscaling is being re-audited against the dimension that actually saturates: sidecar capacity, connection pools, file descriptors, sockets. CPU and request rate were proxies, and agent traffic broke the proxy.
  • Retry and backoff behavior is being treated as a first-class reliability surface, including in client libraries and agent frameworks the platform team does not own.
  • AI as first responder, human as reviewer. Alert triage, dedup, and enrichment increasingly happen on an agent before a human is paged.
  • Postmortem generation is getting automated. Timelines are assembled from chat, telemetry, and deploy logs, and humans edit for judgment and trade-offs instead of reconstructing the chronology by hand.
  • Multi-cloud and exit-plan work got funded after the autumn trifecta. Less full multi-cloud, more “the critical path must survive any one provider.”
  • Hard limits are being treated as tested interfaces. Cloudflare’s oversized-file crash is pushing teams to treat every hard-coded limit as an input boundary that needs a graceful-degradation test.

Takeaways

  1. For every automation, ask how you’ll know it’s misbehaving and how you’ll turn it off without turning off the system.
  2. Anything auto-generated and globally distributed (DNS records, feature files, policies) needs schema validation, size limits that fail gracefully, and staged propagation.
  3. If your incident response now includes AI agents, your postmortem template needs a section for what they did. Blameless applies to models too, but only if you can reconstruct their actions.
  4. Plan for your provider’s bad month, not just their bad hour. October and November 2025 packed a 14-hour, a 50-hour, and a multi-hour global outage into five weeks; July 2026 put four providers on the board in nine days.
  5. Know which of your provider’s features you depend on, not just which provider. CloudFront didn’t fail — VPC Origins did, and only customers using it noticed until the downstream SaaS they relied on went with it. Dependency inventories that stop at the vendor name are half an inventory.
  6. Ask your cloud where the power comes from. europe-west4-a was the window’s longest single outage and had nothing to do with code.
  7. Check what your autoscaler actually measures. If your traffic is increasingly generated by agents rather than people, the shape it was tuned against no longer exists, and the ceiling you hit will be one nobody graphed.
  8. Multi-provider is only a hedge if the providers fail independently. Log every provider failure with a timestamp and review quarterly for clustering. It is crude, but it is the only evidence available to a customer.
  9. Audit the retry behavior of the clients you don’t control. Recovery time is partly a property of somebody else’s SDK.

Sources for this window