Postmortem

Agents On Call (Jul 2025 – Aug 2026) This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them. ...

The CrowdStrike Reckoning (Apr 2024 – Jun 2025) Filed July 19, 2025 — one year to the day after Channel File 291. One Friday in July 2024 caused the largest IT outage in history, and it wasn’t a cloud provider. This window’s postmortems keep circling back to the same theme: other people’s software running inside your trust boundary. A security agent in the kernel, a dealer platform an entire industry depends on, a cloud vendor deleting a customer’s data, a quota policy pushed everywhere at once. ...

The Platform Engineering Pivot (Jan 2023 – Mar 2024) Filed March 8, 2024 — one year after the Datadog incident everyone studied. The big postmortem this window came from an observability vendor going through the exact kind of outage it sells tools to prevent. Meanwhile “platform engineering” started absorbing much of what used to be called DevOps, and the first LLM assistants quietly showed up in incident channels. The incidents that defined the period FAA NOTAM outage, January 2023. A corrupted database file, linked to a contractor’s mistake during maintenance, grounded all US flight departures for hours. It was the first nationwide ground stop since 9/11, and decades-old systems with no hot failover became a topic in Congress. Microsoft Azure WAN, January 25, 2023. A router configuration change (a command that different devices interpreted differently than intended) rippled through Microsoft’s global WAN and broke Azure, Teams, and M365 worldwide for hours. The classic config-change-to-global-outage, at telco scale. Datadog, March 8, 2023. The one everyone studied. An automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes, across multiple cloud providers at the same time (datadoghq.com). Days of degraded service, a reported ~$5M revenue hit, and a thorough multi-part postmortem. Being multi-cloud didn’t help, because the same OS update channel ran across all of them. Redundant copies failed together because they shared a config source. AWS us-east-1, June 13, 2023. A capacity-management issue in Lambda degraded dozens of services for about three hours. Notable admission in the postmortem: AWS’s own support-case system was impaired again. UK air traffic control (NATS), August 2023. A single flight plan with duplicate waypoint names hit an unhandled edge case, and the primary and its identical backup both failed the same way. The independent review became a classic on common-mode software failure. Optus, November 2023. A routing update from an upstream network cascaded into a roughly 14-hour national outage in Australia, affecting emergency calls, and the CEO resigned. Executive accountability for reliability, made explicit. What the postmortems reveal Correlated failure became the top-of-mind risk. Datadog (one update channel across every cloud) and NATS (identical primary and backup software) showed that redundancy without diversity is just bookkeeping. Postmortems started asking which update, config, or code path is shared across the copies you think are independent. ...

When the Map Burns with the Territory (Oct 2021 – Dec 2022) Filed January 11, 2023 — finished, fittingly, on the morning the FAA’s NOTAM outage grounded US departures and opened the next chapter. The image that defines this window: Facebook engineers reportedly couldn’t badge into their own buildings, because the outage had also taken down the door systems. Again and again, the tools meant to fix an outage turned out to depend on the thing that was broken. ...

The us-east-1 Problem (Jul 2020 – Sep 2021) Filed November 25, 2021 — the first anniversary of the Kinesis event, the archetypal us-east-1 story. The incidents in this window share a shape. Something small and deep (a thread limit, a quota system, one customer’s config change) spreads out until half the internet feels it. Readers started asking a new first question: what does everything else quietly depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020. Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit. The fleet needed a slow full restart, and services that depended on it (Cognito, CloudWatch, and various vendors’ status pages) went down with it (aws.amazon.com/message/11201). The postmortem taught a lot of engineers what a cell-based architecture is by describing what it looks like to not have one. Google, December 14, 2020. The identity and quota system took down Gmail, YouTube, and Google Cloud auth for about 47 minutes. An automated quota migration reported usage as zero and starved the auth service. Safety systems that can’t tell “no usage” from “no data” became a recurring theme. Slack, January 4, 2021. First workday of the year. Provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021. A Strasbourg datacenter burned, and some customers learned their backups were in the building that was on fire. Physical DR came back into the conversation. Fastly, June 8, 2021. A dormant bug shipped in May was triggered by one customer’s valid config change and dropped about 85% of Fastly’s network. Global outage in seconds, found in minutes, mostly restored within an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together. Forty-nine minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021. A bug triggered by a config update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal Control plane versus data plane became the sharpest way to look at these. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution: in each case the management machinery failed while the actual capacity was fine. “Static stability,” meaning the data plane keeps working when the control plane is down, became the goal to design toward. ...

One Regex and a Pandemic (Apr 2019 – Jun 2020) Filed July 2, 2020 — one year after the 27-minute regex outage this window is named for. This window has a clean shape. It opens with self-inflicted global outages at Cloudflare and Google that pushed the industry toward staged rollouts, and it closes with COVID-19 stress-testing every capacity plan on the planet. The incidents that defined the period Google Cloud, June 2, 2019. A maintenance automation event descheduled network control-plane jobs across several regions, and the resulting congestion throttled Google Cloud, YouTube, and Gmail for about four hours. The detail everyone remembers: the outage broke the very tools engineers needed to fix it. Cloudflare, July 2, 2019. A single WAF rule with a badly backtracking regex went out globally (WAF rules skipped staged rollout on purpose, for fast response to attacks) and pinned every CPU on Cloudflare’s edge. Twenty-seven minutes of global 502s, and one of the best postmortems ever written (blog.cloudflare.com), including a short lecture on regex complexity and why the kill switch was slow. Verizon BGP route leak, June 24, 2019. A small ISP’s route optimizer leaked routes through Verizon and blackholed chunks of the internet, Cloudflare included. Cloudflare’s blunt writeup (it called the incident “a small heart attack”) pushed RPKI adoption into the mainstream. Stripe, July 2019. Two linked database failures. Stripe published a detailed root-cause report, which was notable for a payments company. Salesforce, May 2019. A database script granted broad permissions across orgs, and the fix (revoking those permissions widely) caused more disruption than the bug. Recovery-as-a-second-incident got a name. COVID-19 surge, March–June 2020. Zoom grew about 30x. Robinhood had repeated trading-day outages from launch-day architecture under thundering-herd load. Streaming services in Europe voluntarily dropped quality, and unemployment systems on mainframes buckled. It was less one incident than a global load test. What the postmortems reveal Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose, for speed against attackers. The lesson wasn’t “never ship fast,” it was “your fastest pipeline needs the strongest safety switches.” Anything that goes global instantly became a flag in design reviews. ...

When Automation Fights Back (Jan 2018 – Mar 2019) Filed April 22, 2019 — the first anniversary of the TSB migration weekend. By 2018, most companies had automated their failover and recovery. The big postmortems this period are about that automation making the wrong call. The question stopped being “why did this break?” and became “why did our own self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018. A big-bang core-banking migration locked UK customers out of their accounts for weeks. The independent review that followed became required reading on cutover risk, and regulators started treating operational resilience as something they could enforce. GitHub, October 21, 2018. A 43-second network partition between the US East and West Coast datacenters led the orchestration software to promote a West Coast MySQL primary while the East Coast primary still held writes that hadn’t replicated. That’s a split-brain. GitHub chose data consistency over uptime, ran degraded for about 24 hours, and published a strong hour-by-hour writeup (github.blog). Microsoft Azure South Central US, September 2018. A lightning strike caused a cooling failure, hardware shut itself down to avoid damage, and the regional outage showed how many “global” Azure services (including Azure AD and the status page) secretly depended on one region. Google Cloud, July 2018. A global load-balancing config change briefly broke customers worldwide, reinforcing a growing theme: global control planes mean a global blast radius. Facebook, March 13, 2019. A server configuration change caused a roughly 14-hour outage of Facebook, Instagram, and WhatsApp, the longest in the company’s history at the time. Wells Fargo, February 2019. A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal Automated failover needs a plan for data. GitHub’s incident became the case study: failover that optimizes for uptime can quietly lose consistency. Postmortems started asking what the orchestrator actually does during a network partition. ...

Typos That Broke the Internet (Oct 2016 – Dec 2017) Filed February 4, 2018 — the weekend after GitLab’s live-streamed database recovery turned one. This is the window that proved honest postmortems build trust instead of costing it. A livestreamed database recovery and a single typo that broke half the web produced two of the most-read incident reports ever written. The incidents that defined the period Dyn DNS DDoS, October 21, 2016. The Mirai botnet, built from hacked IoT devices, took down a big managed-DNS provider and, with it, Twitter, Netflix, Reddit, and GitHub for most of a day. It was the industry’s introduction to dependency concentration: dozens of “independent” sites all used one DNS provider. GitLab, January 31, 2017. An exhausted engineer fighting replication lag ran rm -rf on the primary’s data directory. Five different backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute writeup (about.gitlab.com). About six hours of data was lost, and GitLab’s reputation arguably went up. AWS S3 us-east-1, February 28, 2017. An operator debugging the billing system mistyped a command and removed far more capacity than intended. The index subsystem needed a full restart it hadn’t done in years (aws.amazon.com/message/41926). Thousands of sites broke, including AWS’s own status page, whose health icons were hosted on S3. Cloudbleed, February 2017. A parser bug leaked memory from one Cloudflare customer into other customers’ cached pages. Cloudflare’s detailed disclosure set a new bar for security postmortems. British Airways, May 2017. A datacenter power event grounded flights worldwide. The vague public explanation became the counterexample to GitLab-style openness. Equifax breach, 2017. An unpatched Struts vulnerability. The lesson was less about the bug and more about knowing what you run and keeping it patched. What the postmortems reveal Transparency won, clearly. GitLab and AWS gave specifics: the command, the parameter, the checks they were adding. BA gave vagueness. Customers noticed which companies they trusted more afterward, and “publish the real postmortem” became a strategy rather than a legal risk. ...

The SRE Book Era (Jul 2015 – Sep 2016) Filed September 20, 2016 — one year to the day after the DynamoDB cascade that defined this window. Google published Site Reliability Engineering in 2016 and gave the industry a shared vocabulary: SLOs, error budgets, toil, blameless postmortems. At the same time, the era’s biggest outages kept showing the same pattern. Systems broke not from the original fault, but from how they tried to recover. ...

Shared Fate in the Cloud (Apr 2014 – Jun 2015) Filed July 12, 2015 — the weekend after the NYSE, United Airlines, and the WSJ all fell over on the same Wednesday. This is when everyone learned that moving to the cloud means sharing your provider’s mistakes too: their patches, their config pushes, their operators' typos. Security incidents also started getting the same careful postmortem treatment as plain downtime. The incidents that defined the period Heartbleed, April 2014. The OpenSSL bug that forced mass certificate rotation across the internet. The real operational lesson was that almost nobody had a list of where TLS actually terminated, so “patch and rotate” took weeks. Shellshock in September repeated the drill for bash. Joyent, May 2014. An operator running a routine update rebooted an entire data center of customer systems with one command. Joyent’s postmortem was direct: the problem wasn’t the operator, it was that the tooling let you target a whole datacenter with no confirmation. AWS Xen reboot, September 2014. AWS rebooted a large share of EC2 instances to patch a Xen bug before it was disclosed. Customers who had designed for instance failure barely noticed. Those who hadn’t found out the hard way which servers they’d been treating as pets. Microsoft Azure Storage, November 2014. A performance fix went out globally, skipping the usual staged rollout, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the team had gone around its own rollout policy. It’s one of the most-cited config-change postmortems there is. NYSE, United Airlines, and the WSJ, July 8, 2015. Three unrelated same-day outages that everyone assumed were connected. A good example of how reliability failures turn into a news cycle. What the postmortems reveal Configuration changes became the leading cause. The Azure writeup captured a pattern that still dominates postmortems: the code was fine, and the rollout of a config flag was what broke things. “All deploys are staged, including config” started showing up in action items. ...