Azure on Azarudeen.com

Agents On Call: DNS Races, Feature Files, and the AI-Assisted Postmortem

Tue, 01 Jul 2025 10:00:00 +0530

Agents On Call (Jul 2025 – Jul 2026)

This window opened with a brutal autumn: within a month, AWS, Azure, and Cloudflare each suffered a headline global outage, making “the internet is three companies in a trench coat” a mainstream news take. Meanwhile the biggest practice shift since the SRE book has been underway — AI agents moving from summarizing incidents to responding to them.

The incidents defining the period (so far)

AWS us-east-1, October 20, 2025 — A latent race condition in DynamoDB’s automated DNS management produced an empty DNS record for the regional endpoint; the automation couldn’t self-repair, and failures cascaded through the many AWS services (and thousands of customer apps) that depend on DynamoDB in us-east-1. Roughly 14–15 hours of disruption; Snapchat alone drew ~3 million outage reports. The most consequential us-east-1 event since December 2021 — and an “automation deadlock” case study: the fix required humans to disable the automation that was supposed to prevent exactly this.
Azure Front Door, October 29, 2025 — An inadvertent configuration change broke Microsoft’s global edge/CDN layer for ~8 hours, taking down the Azure portal, M365 entry points, and customer sites — days before earnings, a week after AWS’s turn. A separate East US2 networking config outage lasting roughly 50 hours underlined that regional incidents can now outlast news cycles.
Cloudflare, November 18, 2025 — A database permissions change caused the Bot Management feature file to double in size, exceeding a hard-coded limit in the core proxy; processes crash-looped globally. X, ChatGPT, and Canva threw 5xx errors for hours. Cloudflare’s same-week postmortem (blog.cloudflare.com) echoed their 2019 regex writeup: an internally-generated “content” artifact, globally propagated, hitting an untested limit.
Cloudflare, December 5, 2025 and February 20, 2026 — A ~25-minute traffic outage, then a BGP withdrawal affecting Bring-Your-Own-IP customers — smaller events, but notable for the now-routine speed and detail of disclosure.

(This is a living post, updated through July 2026.)

The Platform Engineering Pivot: Datadog's $5M Lesson and the First AI Whispers

Sun, 01 Jan 2023 10:00:00 +0530

The Platform Engineering Pivot (Jan 2023 – Mar 2024)

This window’s marquee postmortem came from an observability vendor taking its own medicine, while the industry around it reorganized: “platform engineering” absorbed much of DevOps’s identity, and the first LLM assistants quietly joined incident channels.

The incidents that defined the period

FAA NOTAM outage, January 2023 — A corrupted database file (linked to a contractor’s procedural error during maintenance) grounded all US flight departures for hours — the first nationwide ground stop since 9/11. Decades-old systems with no hot failover became a congressional topic.
Microsoft Azure WAN, January 25, 2023 — A router configuration change (a command evaluated differently than intended across devices) rippled through Microsoft’s global WAN, breaking Azure, Teams, and M365 worldwide for hours. Config-change-to-global-blast-radius, the classic, at telco scale.
Datadog, March 8, 2023 — The one everyone studied: an automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes across multiple cloud providers simultaneously (datadoghq.com). Days of degraded service, a reported ~$5M revenue impact, and an exemplary multi-part postmortem. Being multi-cloud didn’t help — the same OS update channel spanned all of them. Correlated failure via configuration management, proven at scale.
AWS us-east-1, June 13, 2023 — A capacity-management issue in Lambda degraded dozens of services for ~3 hours; notable postmortem admission: AWS’s own support-case system was impaired, again.
UK air traffic control (NATS), August 2023 — A single flight plan with duplicate waypoint names hit an unhandled edge case; primary and identical backup failed the same way. The independent review became a classic on common-mode software failure.
Optus, November 2023 — A routing update from an upstream network cascaded into a ~14-hour national outage in Australia (emergency calls affected); the CEO resigned. Executive accountability for reliability, made explicit.

What the postmortems reveal

1. Correlated failure became the top-of-mind risk. Datadog’s incident (one update channel, every cloud) and NATS (identical primary/backup software) showed that redundancy without diversity is bookkeeping. Postmortems began asking: what update, config, or code path is shared across our “independent” copies?

When Automation Fights Back: Split Brains, Lightning Strikes, and SLOs at Scale

Mon, 01 Jan 2018 10:00:00 +0530

When Automation Fights Back (Jan 2018 – Mar 2019)

By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from “why did the component fail?” to “why did our self-healing make it worse?”

The incidents that defined the period

TSB Bank migration, April 2018 — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain.
GitHub, October 21, 2018 — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. Split-brain. GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (github.blog).
Microsoft Azure South Central US, September 2018 — A lightning strike caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many “global” Azure services (including Azure AD and the status portal) had hidden dependencies on one region.
Google Cloud, July 2018 — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius.
Facebook, March 13, 2019 — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a server configuration change — at the time the longest outage in the company’s history.
Wells Fargo, February 2019 — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems.

What the postmortems reveal

1. Automated failover needs a theory of data. GitHub’s incident became the case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking “what does our orchestrator do during a partition?” — a Jepsen-style question applied to ops tooling.