Azure

Agents On Call (Jul 2025 – Aug 2026) This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them. ...

The Platform Engineering Pivot (Jan 2023 – Mar 2024) Filed March 8, 2024 — one year after the Datadog incident everyone studied. The big postmortem this window came from an observability vendor going through the exact kind of outage it sells tools to prevent. Meanwhile “platform engineering” started absorbing much of what used to be called DevOps, and the first LLM assistants quietly showed up in incident channels. The incidents that defined the period FAA NOTAM outage, January 2023. A corrupted database file, linked to a contractor’s mistake during maintenance, grounded all US flight departures for hours. It was the first nationwide ground stop since 9/11, and decades-old systems with no hot failover became a topic in Congress. Microsoft Azure WAN, January 25, 2023. A router configuration change (a command that different devices interpreted differently than intended) rippled through Microsoft’s global WAN and broke Azure, Teams, and M365 worldwide for hours. The classic config-change-to-global-outage, at telco scale. Datadog, March 8, 2023. The one everyone studied. An automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes, across multiple cloud providers at the same time (datadoghq.com). Days of degraded service, a reported ~$5M revenue hit, and a thorough multi-part postmortem. Being multi-cloud didn’t help, because the same OS update channel ran across all of them. Redundant copies failed together because they shared a config source. AWS us-east-1, June 13, 2023. A capacity-management issue in Lambda degraded dozens of services for about three hours. Notable admission in the postmortem: AWS’s own support-case system was impaired again. UK air traffic control (NATS), August 2023. A single flight plan with duplicate waypoint names hit an unhandled edge case, and the primary and its identical backup both failed the same way. The independent review became a classic on common-mode software failure. Optus, November 2023. A routing update from an upstream network cascaded into a roughly 14-hour national outage in Australia, affecting emergency calls, and the CEO resigned. Executive accountability for reliability, made explicit. What the postmortems reveal Correlated failure became the top-of-mind risk. Datadog (one update channel across every cloud) and NATS (identical primary and backup software) showed that redundancy without diversity is just bookkeeping. Postmortems started asking which update, config, or code path is shared across the copies you think are independent. ...

When Automation Fights Back (Jan 2018 – Mar 2019) Filed April 22, 2019 — the first anniversary of the TSB migration weekend. By 2018, most companies had automated their failover and recovery. The big postmortems this period are about that automation making the wrong call. The question stopped being “why did this break?” and became “why did our own self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018. A big-bang core-banking migration locked UK customers out of their accounts for weeks. The independent review that followed became required reading on cutover risk, and regulators started treating operational resilience as something they could enforce. GitHub, October 21, 2018. A 43-second network partition between the US East and West Coast datacenters led the orchestration software to promote a West Coast MySQL primary while the East Coast primary still held writes that hadn’t replicated. That’s a split-brain. GitHub chose data consistency over uptime, ran degraded for about 24 hours, and published a strong hour-by-hour writeup (github.blog). Microsoft Azure South Central US, September 2018. A lightning strike caused a cooling failure, hardware shut itself down to avoid damage, and the regional outage showed how many “global” Azure services (including Azure AD and the status page) secretly depended on one region. Google Cloud, July 2018. A global load-balancing config change briefly broke customers worldwide, reinforcing a growing theme: global control planes mean a global blast radius. Facebook, March 13, 2019. A server configuration change caused a roughly 14-hour outage of Facebook, Instagram, and WhatsApp, the longest in the company’s history at the time. Wells Fargo, February 2019. A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal Automated failover needs a plan for data. GitHub’s incident became the case study: failover that optimizes for uptime can quietly lose consistency. Postmortems started asking what the orchestrator actually does during a network partition. ...