The us-east-1 Problem: Control Planes, Quotas, and a 49-Second CDN Outage
The us-east-1 Problem (Jul 2020 – Sep 2021) Filed November 25, 2021 — the first anniversary of the Kinesis event, the archetypal us-east-1 story. The incidents in this window share a shape. Something small and deep (a thread limit, a quota system, one customer’s config change) spreads out until half the internet feels it. Readers started asking a new first question: what does everything else quietly depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020. Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit. The fleet needed a slow full restart, and services that depended on it (Cognito, CloudWatch, and various vendors’ status pages) went down with it (aws.amazon.com/message/11201). The postmortem taught a lot of engineers what a cell-based architecture is by describing what it looks like to not have one. Google, December 14, 2020. The identity and quota system took down Gmail, YouTube, and Google Cloud auth for about 47 minutes. An automated quota migration reported usage as zero and starved the auth service. Safety systems that can’t tell “no usage” from “no data” became a recurring theme. Slack, January 4, 2021. First workday of the year. Provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021. A Strasbourg datacenter burned, and some customers learned their backups were in the building that was on fire. Physical DR came back into the conversation. Fastly, June 8, 2021. A dormant bug shipped in May was triggered by one customer’s valid config change and dropped about 85% of Fastly’s network. Global outage in seconds, found in minutes, mostly restored within an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together. Forty-nine minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021. A bug triggered by a config update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal Control plane versus data plane became the sharpest way to look at these. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution: in each case the management machinery failed while the actual capacity was fine. “Static stability,” meaning the data plane keeps working when the control plane is down, became the goal to design toward. ...