Faa

The Platform Engineering Pivot (Jan 2023 – Mar 2024) This window’s marquee postmortem came from an observability vendor taking its own medicine, while the industry around it reorganized: “platform engineering” absorbed much of DevOps’s identity, and the first LLM assistants quietly joined incident channels. The incidents that defined the period FAA NOTAM outage, January 2023 — A corrupted database file (linked to a contractor’s procedural error during maintenance) grounded all US flight departures for hours — the first nationwide ground stop since 9/11. Decades-old systems with no hot failover became a congressional topic. Microsoft Azure WAN, January 25, 2023 — A router configuration change (a command evaluated differently than intended across devices) rippled through Microsoft’s global WAN, breaking Azure, Teams, and M365 worldwide for hours. Config-change-to-global-blast-radius, the classic, at telco scale. Datadog, March 8, 2023 — The one everyone studied: an automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes across multiple cloud providers simultaneously (datadoghq.com). Days of degraded service, a reported ~$5M revenue impact, and an exemplary multi-part postmortem. Being multi-cloud didn’t help — the same OS update channel spanned all of them. Correlated failure via configuration management, proven at scale. AWS us-east-1, June 13, 2023 — A capacity-management issue in Lambda degraded dozens of services for ~3 hours; notable postmortem admission: AWS’s own support-case system was impaired, again. UK air traffic control (NATS), August 2023 — A single flight plan with duplicate waypoint names hit an unhandled edge case; primary and identical backup failed the same way. The independent review became a classic on common-mode software failure. Optus, November 2023 — A routing update from an upstream network cascaded into a ~14-hour national outage in Australia (emergency calls affected); the CEO resigned. Executive accountability for reliability, made explicit. What the postmortems reveal 1. Correlated failure became the top-of-mind risk. Datadog’s incident (one update channel, every cloud) and NATS (identical primary/backup software) showed that redundancy without diversity is bookkeeping. Postmortems began asking: what update, config, or code path is shared across our “independent” copies? ...