Postmortem on Azarudeen.com

Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Tue, 01 Jul 2025 10:00:00 +0530

Agents On Call (Jul 2025 – Jul 2026)

This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them.

The incidents defining the period (so far)

AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this.
Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles.
Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit.
Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure.

(This is a living post, updated through July 2026.)

The CrowdStrike Reckoning: Third-Party Risk Becomes Everyone's Root Cause

Mon, 01 Apr 2024 10:00:00 +0530

The CrowdStrike Reckoning (Apr 2024 – Jun 2025)

One Friday in July 2024 produced the largest IT outage in history — and it wasn’t a cloud provider. This window’s postmortems are dominated by other people’s software running inside your trust boundary: security agents in the kernel, a dealer platform for an entire industry, a cloud vendor deleting a customer, and a quota policy pushed worldwide.

The incidents that defined the period

Google Cloud / UniSuper, May 2024 — A misconfiguration during provisioning led Google Cloud to delete an entire customer’s private cloud subscription — a ~$125B pension fund — causing ~two weeks of disruption. Recovery leaned on UniSuper’s own third-party backups. The joint apology statement was unprecedented; “what if our cloud account itself is the failure domain?” entered every DR review.
CDK Global, June 2024 — Ransomware took down the SaaS platform underpinning ~15,000 North American car dealerships for weeks. A whole industry discovered it had a single point of failure it had never load-tested: its vendor.
CrowdStrike, July 19, 2024 — A faulty Rapid Response Content update (Channel File 291) hit an out-of-bounds read in the Falcon sensor running in the Windows kernel: ~8.5 million machines blue-screened. Airlines, hospitals, banks, 911 centers. Insured losses in the billions; Delta alone claimed ~$500M. The RCA and Congressional testimony detailed the gap: sensor code was staged and tested; content updates were validated by a checker with a bug and deployed globally at once (crowdstrike.com RCA).
Azure Central US, July 30, 2024 — A DDoS defense misconfiguration amplified rather than mitigated an attack, in a summer of repeated Microsoft incidents.
OpenAI, December 11, 2024 — A new telemetry service overwhelmed Kubernetes API servers across clusters; DNS caching masked the rollout risk, and engineers were locked out of the control planes they needed to revert. A modern classic: observability tooling as the outage trigger, published with unusual candor for an AI lab.
Google Cloud, June 12, 2025 — A new Service Control policy feature with a null-pointer path, no feature flag, and instant global metadata replication crash-looped API management worldwide (~3 hours; ~7.5h for us-central1). Cloudflare (whose Workers KV depended on GCS), Spotify, and dozens of others went down with it. The postmortem’s own action items read like this series' greatest hits: flag-gate everything, stagger global propagation, add backoff.

What the postmortems reveal

1. “Content” updates are code. CrowdStrike’s split — rigorous staging for binaries, instant global push for configuration content — is the same pattern as Cloudflare 2019 and Google 2025. The industry’s hardest-won lesson keeps recurring one abstraction level up: anything that changes runtime behavior needs canaries, whether it’s called code, config, content, or policy.

The Platform Engineering Pivot: Datadog's $5M Lesson and the First AI Whispers

Sun, 01 Jan 2023 10:00:00 +0530

The Platform Engineering Pivot (Jan 2023 – Mar 2024)

This window’s marquee postmortem came from an observability vendor taking its own medicine, while the industry around it reorganized: “platform engineering” absorbed much of DevOps’s identity, and the first LLM assistants quietly joined incident channels.

The incidents that defined the period

FAA NOTAM outage, January 2023 — A corrupted database file (linked to a contractor’s procedural error during maintenance) grounded all US flight departures for hours — the first nationwide ground stop since 9/11. Decades-old systems with no hot failover became a congressional topic.
Microsoft Azure WAN, January 25, 2023 — A router configuration change (a command evaluated differently than intended across devices) rippled through Microsoft’s global WAN, breaking Azure, Teams, and M365 worldwide for hours. Config-change-to-global-blast-radius, the classic, at telco scale.
Datadog, March 8, 2023 — The one everyone studied: an automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes across multiple cloud providers simultaneously (datadoghq.com). Days of degraded service, a reported ~$5M revenue impact, and an exemplary multi-part postmortem. Being multi-cloud didn’t help — the same OS update channel spanned all of them. Correlated failure via configuration management, proven at scale.
AWS us-east-1, June 13, 2023 — A capacity-management issue in Lambda degraded dozens of services for ~3 hours; notable postmortem admission: AWS’s own support-case system was impaired, again.
UK air traffic control (NATS), August 2023 — A single flight plan with duplicate waypoint names hit an unhandled edge case; primary and identical backup failed the same way. The independent review became a classic on common-mode software failure.
Optus, November 2023 — A routing update from an upstream network cascaded into a ~14-hour national outage in Australia (emergency calls affected); the CEO resigned. Executive accountability for reliability, made explicit.

What the postmortems reveal

1. Correlated failure became the top-of-mind risk. Datadog’s incident (one update channel, every cloud) and NATS (identical primary/backup software) showed that redundancy without diversity is bookkeeping. Postmortems began asking: what update, config, or code path is shared across our “independent” copies?

When the Map Burns with the Territory: BGP Lockouts and Cascading Dependencies

Fri, 01 Oct 2021 10:00:00 +0530

When the Map Burns with the Territory (Oct 2021 – Dec 2022)

The defining image of this window is Facebook engineers reportedly unable to badge into their own buildings because the outage had taken down the systems that controlled the doors. Incident after incident showed recovery tooling, communications, and even physical access welded to the infrastructure they were supposed to repair.

The incidents that defined the period

Facebook/Meta, October 4, 2021 — A routine maintenance command disconnected Facebook’s backbone; its DNS servers, by design, withdrew their BGP routes when they couldn’t reach the datacenters. Facebook, Instagram, and WhatsApp vanished from the internet for ~6 hours. Internal tools and remote access died too, forcing physical datacenter visits (engineering.fb.com).
Roblox, October 28–31, 2021 — A 73-hour outage from a subtle interaction between a Consul feature and BoltDB performance. The postmortem, co-published with HashiCorp months later, was praised for depth and for neither party hiding behind the other.
AWS us-east-1, December 7, 2021 — An automated scaling activity triggered a thundering herd on the internal network connecting AWS’s own services; monitoring and support tooling were among the casualties, slowing diagnosis (aws.amazon.com/message/12721). Two further December us-east-1 incidents made “why is everything in one region?” a CTO-level question.
Log4Shell, December 2021 — A logging library CVE that turned every Java shop’s December into an incident. The response was run like an outage and postmortem’d like one; SBOMs went from acronym to mandate.
Atlassian, April 2022 — A maintenance script given the wrong IDs permanently deleted ~400 customers’ cloud sites; restoration took up to two weeks because recovery was designed for whole-service rollback, not per-customer restore. The postmortem’s candor about that gap was the lesson.
Rogers, July 8, 2022 — A maintenance update removed a routing filter and the resulting BGP flood crashed Canada’s largest network — including 911 access and Interac payments — for ~a day. National reviews followed; reliability became telecom regulation.
Cloudflare, June 21, 2022 — A BGP change during a datacenter conversion took down 19 of their busiest locations; postmortem published same day.
UK heatwave, July 2022 — Google and Oracle cloud regions in London throttled by cooling failures: climate as a reliability factor.
Southwest Airlines, December 2022 — Crew-scheduling software collapsed under a winter storm; ~17,000 flights cancelled. The eventual reckoning (including a record fine) made “legacy system risk” a board agenda item.

What the postmortems reveal

1. Recovery must not depend on the thing being recovered. Facebook’s DNS, AWS’s monitoring, Atlassian’s restore tooling — each incident extended because the repair path ran through the failure. Out-of-band management networks, break-glass access, and offline runbooks became the era’s universal action item.

The us-east-1 Problem: Control Planes, Quotas, and a 49-Second CDN Outage

Wed, 01 Jul 2020 10:00:00 +0530

The us-east-1 Problem (Jul 2020 – Sep 2021)

The incidents of this window share a shape: a small, deep dependency — a thread limit, a quota system, one customer’s config — radiating outward until half the internet notices. Postmortem readers learned to ask a new first question: what does everything else depend on?

The incidents that defined the period

AWS Kinesis / us-east-1, November 25, 2020 — Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit; the fleet needed a slow full restart, and dependent services (Cognito, CloudWatch — and vendors' status pages) failed with it (aws.amazon.com/message/11201). The postmortem taught thousands of engineers what a cell-based architecture is — by describing its absence.
Google, December 14, 2020 — The identity/quota system took down Gmail, YouTube, and Google Cloud auth for ~47 minutes: an automated quota migration reported usage as zero and rationed the auth service to death. Safety systems that can’t distinguish “no usage” from “no data” became a postmortem archetype.
Slack, January 4, 2021 — First workday of the year; provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering).
OVHcloud fire, March 2021 — A Strasbourg datacenter burned; some customers learned their “backups” lived in the building that was on fire. Physical DR returned to the conversation.
Fastly, June 8, 2021 — A dormant bug shipped in May was triggered by one customer’s valid configuration change, dropping ~85% of Fastly’s network. Global outage in seconds; identified in minutes; largely restored in under an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together — 49 minutes that made “CDN concentration” a mainstream news topic.
Akamai Edge DNS, July 2021 — A bug triggered by a configuration update took down banks and airlines for about an hour. Same lesson, different CDN.

What the postmortems reveal

1. Control plane vs data plane became the sharpest lens. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution — in each case the management machinery failed while the underlying capacity was fine. “Static stability” (the data plane keeps working when the control plane is down) became the design goal to cite.

One Regex and a Pandemic: Global Blast Radius Meets Global Load

Mon, 01 Apr 2019 10:00:00 +0530

One Regex and a Pandemic (Apr 2019 – Jun 2020)

This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth.

The incidents that defined the period

Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage.
Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow.
Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream.
Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company.
Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern.
COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test.

What the postmortems reveal

1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews.

When Automation Fights Back: Split Brains, Lightning Strikes, and SLOs at Scale

Mon, 01 Jan 2018 10:00:00 +0530

When Automation Fights Back (Jan 2018 – Mar 2019)

By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from “why did the component fail?” to “why did our self-healing make it worse?”

The incidents that defined the period

TSB Bank migration, April 2018 — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain.
GitHub, October 21, 2018 — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. Split-brain. GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (github.blog).
Microsoft Azure South Central US, September 2018 — A lightning strike caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many “global” Azure services (including Azure AD and the status portal) had hidden dependencies on one region.
Google Cloud, July 2018 — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius.
Facebook, March 13, 2019 — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a server configuration change — at the time the longest outage in the company’s history.
Wells Fargo, February 2019 — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems.

What the postmortems reveal

1. Automated failover needs a theory of data. GitHub’s incident became the case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking “what does our orchestrator do during a partition?” — a Jepsen-style question applied to ops tooling.

Typos That Broke the Internet: S3, GitLab, and Radical Transparency

Sat, 01 Oct 2016 10:00:00 +0530

Typos That Broke the Internet (Oct 2016 – Dec 2017)

If one window proved that public, honest postmortems build more trust than they cost, it’s this one. A livestreamed database recovery and a typo that took down half the web produced two of the most-read incident reports in history.

The incidents that defined the period

Dyn DNS DDoS, October 21, 2016 — The Mirai botnet, built from IoT devices, took down a major managed-DNS provider and with it Twitter, Netflix, Reddit, and GitHub for much of a day. The industry’s introduction to dependency concentration: dozens of “independent” sites shared one DNS provider.
GitLab database incident, January 31, 2017 — An exhausted engineer, fighting replication lag, ran rm -rf on the primary’s data directory. Five backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute postmortem (about.gitlab.com). ~6 hours of data was lost — and GitLab’s reputation arguably improved.
AWS S3 us-east-1, February 28, 2017 — An operator debugging the billing system mistyped a playbook parameter and removed far more capacity than intended; the index subsystem required a full restart it hadn’t had in years (aws.amazon.com/message/41926). Thousands of sites broke — including, memorably, AWS’s own status page, whose health icons were hosted on S3.
Cloudbleed, February 2017 — A parser bug leaked memory across Cloudflare customers into cached pages. Cloudflare’s forensic-grade disclosure set a new bar for security postmortems.
British Airways, May 2017 — A datacenter power event (a contractor and a UPS) grounded flights globally; the vague public explanation became the counterexample to GitLab-style transparency.
Equifax breach, 2017 — An unpatched Struts vulnerability; the postmortem lesson was less about the bug than about asset inventory and patch governance.

What the postmortems reveal

1. Transparency won, decisively. GitLab and AWS gave specifics (the command, the parameter, the safety checks now added); BA gave vagueness. The market noticed which companies it trusted more afterward. “Publish the real postmortem” became a competitive strategy, not a legal risk.

The SRE Book Era: Error Budgets Meet Cascading Failure

Wed, 01 Jul 2015 10:00:00 +0530

The SRE Book Era (Jul 2015 – Sep 2016)

Google published Site Reliability Engineering in April 2016 and handed the industry a shared vocabulary: SLOs, error budgets, toil, and a whole chapter on postmortem culture. Meanwhile, the period’s biggest incidents were masterclasses in cascading failure — systems that fell over not from the initial fault, but from their own recovery behavior.

The incidents that defined the period

AWS DynamoDB, September 20, 2015 — The canonical cascading-failure postmortem (aws.amazon.com/message/5467D2). A network disruption caused storage servers to re-request membership metadata simultaneously; the metadata service, already near capacity from a new index feature, couldn’t serve the herd; retries made it worse. DynamoDB’s outage cascaded into EC2, SQS, and CloudWatch in us-east-1. Action items — capacity headroom, longer timeouts, segmented retries — read like a distributed-systems syllabus.
Salesforce NA14, May 2016 — A database failure plus a failed failover left a major instance degraded for nearly a day, with some data unrecoverable. It pushed “your SaaS vendor’s DR plan is your DR plan” into procurement conversations.
Southwest Airlines (July 2016) and Delta (August 2016) — Back-to-back airline meltdowns from single-point-of-failure infrastructure (a failed router; a datacenter power incident) cancelling thousands of flights. Boards started asking about technical debt.
Telstra, 2016 — A string of national mobile outages in Australia, one triggered by a single node being taken offline incorrectly, normalized the telco postmortem press release.

What the postmortems reveal

1. Retry storms became a named enemy. The DynamoDB writeup made “metastable failure” patterns mainstream years before the academic term: exponential backoff, jitter, circuit breakers, and load shedding moved from Netflix blog posts into default library behavior (and into everyone’s action items).

Shared Fate: Heartbleed, Mass Reboots, and the Limits of Cloud Trust

Tue, 01 Apr 2014 10:00:00 +0530

Shared Fate in the Cloud (Apr 2014 – Jun 2015)

This window is when the industry learned that moving to the cloud means sharing your provider’s fate — their hypervisor patches, their config rollouts, and their operators’ keystrokes. It’s also when security incidents started being written up with the same discipline as availability incidents.

The incidents that defined the period

Heartbleed, April 2014 — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so “patch and rotate” took weeks. Shellshock (September 2014) repeated the drill for bash.
Joyent, May 2014 — An operator running a routine update rebooted an entire data center’s worth of customer systems with one command. Joyent’s postmortem was admirably direct: the problem wasn’t the operator, it was that the tooling allowed a datacenter-wide target with no confirmation. A textbook blameless writeup.
AWS Xen reboot, September 2014 — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the “design for instance failure” gospel (Netflix, famously) sailed through; those who hadn’t discovered pet servers the hard way.
Microsoft Azure Storage, November 2014 — A performance fix was rolled out globally, skipping the staged “flighting” process, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever.
NYSE, United Airlines, and the WSJ — July 8, 2015 — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles.

What the postmortems reveal

1. Configuration change became the leading villain. The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the rollout of a config flag was the failure. “All deploys are staged, no exceptions, including configuration” started appearing in action items.

The Blameless Awakening: How Postmortems Became Engineering Culture

Tue, 01 Jan 2013 10:00:00 +0530

The Blameless Awakening (Jan 2013 – Mar 2014)

In early 2013, the public postmortem was still a novelty. Most companies treated outages as PR problems to be minimized, not learning opportunities to be shared. By the spring of 2014, that had visibly changed — and this 15-month window is where the modern postmortem culture took root.

The incidents that defined the period

Microsoft Azure, February 2013 — A worldwide Azure Storage outage caused by an expired SSL certificate. The lesson — certificate lifecycle management is an operational discipline, not a checkbox — still gets relearned every year.
Google, August 2013 — Google went dark for roughly 2–5 minutes, and global internet traffic reportedly dropped ~40%. The first mainstream glimpse of how concentrated the web had become.
Amazon.com, August 2013 — A ~30-minute outage of the retail site, widely used to popularize “downtime costs $X per minute” math in reliability business cases.
NASDAQ “Flash Freeze,” August 2013 — A three-hour trading halt traced to a software flaw in the Securities Information Processor, showing that finance’s bespoke infrastructure had the same failure modes as web systems.
HealthCare.gov, October 2013 — Not a cloud outage but the era’s defining systems failure: a big-bang launch with no load testing, no incremental rollout, and no operational ownership. Its rescue by a small team of web-industry engineers seeded what later became the US Digital Service — and became the canonical argument for DevOps practices in government and enterprise.

What the postmortems reveal

1. “Blameless” went from Etsy blog post to industry norm. John Allspaw’s writing on blameless postmortems and Etsy’s open-sourced Morgue tool (their internal postmortem tracker) gave teams both the philosophy and the software. The core idea — engineers closest to the failure have the most information, and punishing them destroys that information — started appearing in conference talks everywhere.