The Platform Engineering Pivot (Jan 2023 – Mar 2024)

Filed March 8, 2024 — one year after the Datadog incident everyone studied.

The big postmortem this window came from an observability vendor going through the exact kind of outage it sells tools to prevent. Meanwhile “platform engineering” started absorbing much of what used to be called DevOps, and the first LLM assistants quietly showed up in incident channels.

The incidents that defined the period

  • FAA NOTAM outage, January 2023. A corrupted database file, linked to a contractor’s mistake during maintenance, grounded all US flight departures for hours. It was the first nationwide ground stop since 9/11, and decades-old systems with no hot failover became a topic in Congress.
  • Microsoft Azure WAN, January 25, 2023. A router configuration change (a command that different devices interpreted differently than intended) rippled through Microsoft’s global WAN and broke Azure, Teams, and M365 worldwide for hours. The classic config-change-to-global-outage, at telco scale.
  • Datadog, March 8, 2023. The one everyone studied. An automatic security update to systemd across their fleet triggered a network stack reset on tens of thousands of nodes, across multiple cloud providers at the same time (datadoghq.com). Days of degraded service, a reported ~$5M revenue hit, and a thorough multi-part postmortem. Being multi-cloud didn’t help, because the same OS update channel ran across all of them. Redundant copies failed together because they shared a config source.
  • AWS us-east-1, June 13, 2023. A capacity-management issue in Lambda degraded dozens of services for about three hours. Notable admission in the postmortem: AWS’s own support-case system was impaired again.
  • UK air traffic control (NATS), August 2023. A single flight plan with duplicate waypoint names hit an unhandled edge case, and the primary and its identical backup both failed the same way. The independent review became a classic on common-mode software failure.
  • Optus, November 2023. A routing update from an upstream network cascaded into a roughly 14-hour national outage in Australia, affecting emergency calls, and the CEO resigned. Executive accountability for reliability, made explicit.

What the postmortems reveal

Correlated failure became the top-of-mind risk. Datadog (one update channel across every cloud) and NATS (identical primary and backup software) showed that redundancy without diversity is just bookkeeping. Postmortems started asking which update, config, or code path is shared across the copies you think are independent.

Unattended automation earned real change management. The systemd update wasn’t a deploy anyone approved that morning. Fleet-wide auto-updates, autoscalers, and cert renewals started getting the same staging, canaries, and pause buttons as human-initiated changes.

Postmortems became products. Datadog published deep-dive series, and Azure moved to structured incident reviews with named review calls for big incidents. The audience was no longer just customers; it was regulators, analysts, and recruiting.

LLMs entered the incident channel. By late 2023, teams were trying ChatGPT and Claude for summarizing incident timelines and drafting postmortems. It was assistive, not autonomous, but the direction was set.

Practice and tooling shifts

  • Platform engineering (internal developer platforms, golden paths) rebranded much of DevOps, and reliability became a platform feature rather than a per-team heroic effort.
  • DORA metrics and SLOs merged into standard executive reporting.
  • Incident tooling consolidated around Slack-native workflows with auto-generated timelines, which are exactly the raw material for AI summarization.

Takeaways that still hold

  1. Redundancy only counts if the copies can’t fail the same way. Audit shared update channels, base images, and config sources.
  2. Every automated change path needs staging, canaries, and a pause button, especially the ones no human triggers.
  3. Fuzz your inputs. One malformed flight plan (or feature file) finding an unhandled edge case in both primary and backup is a pattern, not a fluke.