The Platform Engineering Pivot (Jan 2023 – Mar 2024)
This window’s marquee postmortem came from an observability vendor taking its own medicine, while the industry around it reorganized: “platform engineering” absorbed much of DevOps’s identity, and the first LLM assistants quietly joined incident channels.
The incidents that defined the period
- FAA NOTAM outage, January 2023 — A corrupted database file (linked to a contractor’s procedural error during maintenance) grounded all US flight departures for hours — the first nationwide ground stop since 9/11. Decades-old systems with no hot failover became a congressional topic.
- Microsoft Azure WAN, January 25, 2023 — A router configuration change (a command evaluated differently than intended across devices) rippled through Microsoft’s global WAN, breaking Azure, Teams, and M365 worldwide for hours. Config-change-to-global-blast-radius, the classic, at telco scale.
- Datadog, March 8, 2023 — The one everyone studied: an automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes across multiple cloud providers simultaneously (datadoghq.com). Days of degraded service, a reported ~$5M revenue impact, and an exemplary multi-part postmortem. Being multi-cloud didn’t help — the same OS update channel spanned all of them. Correlated failure via configuration management, proven at scale.
- AWS us-east-1, June 13, 2023 — A capacity-management issue in Lambda degraded dozens of services for ~3 hours; notable postmortem admission: AWS’s own support-case system was impaired, again.
- UK air traffic control (NATS), August 2023 — A single flight plan with duplicate waypoint names hit an unhandled edge case; primary and identical backup failed the same way. The independent review became a classic on common-mode software failure.
- Optus, November 2023 — A routing update from an upstream network cascaded into a ~14-hour national outage in Australia (emergency calls affected); the CEO resigned. Executive accountability for reliability, made explicit.
What the postmortems reveal
1. Correlated failure became the top-of-mind risk. Datadog’s incident (one update channel, every cloud) and NATS (identical primary/backup software) showed that redundancy without diversity is bookkeeping. Postmortems began asking: what update, config, or code path is shared across our “independent” copies?
2. Unattended automation earned change-management. The systemd update wasn’t a deploy anyone approved that morning. Fleet-wide auto-updates, autoscalers, and cert renewals started getting the same staging, canarying, and pause buttons as human-initiated changes.
3. Postmortems became products. Datadog published deep-dive series; Azure moved to structured PIRs with named review calls for large incidents. The audience was no longer just customers — it was regulators, analysts, and recruiting.
4. LLMs entered the incident channel. By late 2023, teams were piloting ChatGPT/Claude for summarizing incident timelines and drafting postmortems. The trend was assistive, not autonomous — but the direction was set.
Practice and tooling shifts
- Platform engineering (internal developer platforms, golden paths) rebranded much of DevOps; reliability became a platform feature rather than a per-team heroic.
- DORA metrics and SLOs converged into standard executive reporting.
- Incident tooling consolidated around Slack-native workflows with auto-generated timelines — the raw material for AI summarization.
Takeaways that still hold
- Redundancy only counts if the copies can’t fail the same way; audit shared update channels, base images, and config sources.
- Every automated change path needs staging, canaries, and a pause button — especially the ones no human triggers.
- Fuzz your inputs: one malformed flight plan (or feature file) finding an unhandled edge case in primary and backup is a pattern, not an anomaly.