http://azarudeen.com/tags/faa/Recent content in Faa on Azarudeen.comHugoen-usSun, 01 Jan 2023 10:00:00 +0530http://azarudeen.com/posts/09-jan-2023-to-mar-2024-the-platform-engineering-pivot/Sun, 01 Jan 2023 10:00:00 +0530http://azarudeen.com/posts/09-jan-2023-to-mar-2024-the-platform-engineering-pivot/<h1 id="the-platform-engineering-pivot-jan-2023--mar-2024">The Platform Engineering Pivot (Jan 2023 – Mar 2024)</h1> <p>This window&rsquo;s marquee postmortem came from an observability vendor taking its own medicine, while the industry around it reorganized: &ldquo;platform engineering&rdquo; absorbed much of DevOps&rsquo;s identity, and the first LLM assistants quietly joined incident channels.</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>FAA NOTAM outage, January 2023</strong> — A corrupted database file (linked to a contractor&rsquo;s procedural error during maintenance) grounded all US flight departures for hours — the first nationwide ground stop since 9/11. Decades-old systems with no hot failover became a congressional topic.</li> <li><strong>Microsoft Azure WAN, January 25, 2023</strong> — A <strong>router configuration change</strong> (a command evaluated differently than intended across devices) rippled through Microsoft&rsquo;s global WAN, breaking Azure, Teams, and M365 worldwide for hours. Config-change-to-global-blast-radius, the classic, at telco scale.</li> <li><strong>Datadog, March 8, 2023</strong> — The one everyone studied: an <strong>automatic security update to systemd</strong> across their fleet triggered a network stack reset on tens of thousands of nodes across <strong>multiple cloud providers simultaneously</strong> (<a href="https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/">datadoghq.com</a>). Days of degraded service, a reported ~$5M revenue impact, and an exemplary multi-part postmortem. Being multi-cloud didn&rsquo;t help — the <em>same OS update channel</em> spanned all of them. Correlated failure via configuration management, proven at scale.</li> <li><strong>AWS us-east-1, June 13, 2023</strong> — A capacity-management issue in Lambda degraded dozens of services for ~3 hours; notable postmortem admission: AWS&rsquo;s own support-case system was impaired, again.</li> <li><strong>UK air traffic control (NATS), August 2023</strong> — A single flight plan with duplicate waypoint names hit an unhandled edge case; primary <em>and</em> identical backup failed the same way. The independent review became a classic on common-mode software failure.</li> <li><strong>Optus, November 2023</strong> — A routing update from an upstream network cascaded into a ~14-hour national outage in Australia (emergency calls affected); the CEO resigned. Executive accountability for reliability, made explicit.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. Correlated failure became the top-of-mind risk.</strong> Datadog&rsquo;s incident (one update channel, every cloud) and NATS (identical primary/backup software) showed that redundancy without <em>diversity</em> is bookkeeping. Postmortems began asking: what update, config, or code path is shared across our &ldquo;independent&rdquo; copies?</p>