The us-east-1 Problem: Control Planes, Quotas, and a 49-Second CDN Outage
The us-east-1 Problem (Jul 2020 – Sep 2021) The incidents of this window share a shape: a small, deep dependency — a thread limit, a quota system, one customer’s config — radiating outward until half the internet notices. Postmortem readers learned to ask a new first question: what does everything else depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020 — Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit; the fleet needed a slow full restart, and dependent services (Cognito, CloudWatch — and vendors' status pages) failed with it (aws.amazon.com/message/11201). The postmortem taught thousands of engineers what a cell-based architecture is — by describing its absence. Google, December 14, 2020 — The identity/quota system took down Gmail, YouTube, and Google Cloud auth for ~47 minutes: an automated quota migration reported usage as zero and rationed the auth service to death. Safety systems that can’t distinguish “no usage” from “no data” became a postmortem archetype. Slack, January 4, 2021 — First workday of the year; provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021 — A Strasbourg datacenter burned; some customers learned their “backups” lived in the building that was on fire. Physical DR returned to the conversation. Fastly, June 8, 2021 — A dormant bug shipped in May was triggered by one customer’s valid configuration change, dropping ~85% of Fastly’s network. Global outage in seconds; identified in minutes; largely restored in under an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together — 49 minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021 — A bug triggered by a configuration update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal 1. Control plane vs data plane became the sharpest lens. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution — in each case the management machinery failed while the underlying capacity was fine. “Static stability” (the data plane keeps working when the control plane is down) became the design goal to cite. ...