The us-east-1 Problem (Jul 2020 – Sep 2021)
Filed November 25, 2021 — the first anniversary of the Kinesis event, the archetypal us-east-1 story.
The incidents in this window share a shape. Something small and deep (a thread limit, a quota system, one customer’s config change) spreads out until half the internet feels it. Readers started asking a new first question: what does everything else quietly depend on?
The incidents that defined the period
- AWS Kinesis / us-east-1, November 25, 2020. Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit. The fleet needed a slow full restart, and services that depended on it (Cognito, CloudWatch, and various vendors’ status pages) went down with it (aws.amazon.com/message/11201). The postmortem taught a lot of engineers what a cell-based architecture is by describing what it looks like to not have one.
- Google, December 14, 2020. The identity and quota system took down Gmail, YouTube, and Google Cloud auth for about 47 minutes. An automated quota migration reported usage as zero and starved the auth service. Safety systems that can’t tell “no usage” from “no data” became a recurring theme.
- Slack, January 4, 2021. First workday of the year. Provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering).
- OVHcloud fire, March 2021. A Strasbourg datacenter burned, and some customers learned their backups were in the building that was on fire. Physical DR came back into the conversation.
- Fastly, June 8, 2021. A dormant bug shipped in May was triggered by one customer’s valid config change and dropped about 85% of Fastly’s network. Global outage in seconds, found in minutes, mostly restored within an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together. Forty-nine minutes that made “CDN concentration” a mainstream news topic.
- Akamai Edge DNS, July 2021. A bug triggered by a config update took down banks and airlines for about an hour. Same lesson, different CDN.
What the postmortems reveal
Control plane versus data plane became the sharpest way to look at these. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution: in each case the management machinery failed while the actual capacity was fine. “Static stability,” meaning the data plane keeps working when the control plane is down, became the goal to design toward.
Concentration risk got measured. After Fastly and Akamai, analysts started publishing how much of the internet shares each provider. Multi-CDN and multi-region went from a cost-conscious “no” to a board-level “why not?”
Monitoring’s circular dependency got exposed. AWS’s status updates, vendors' status pages hosted on the failing cloud, and Slack’s own dashboards all degraded along with the incident. “Monitor from outside your own failure domain” showed up in action items across the industry.
Cold starts are the hardest path. The Kinesis fleet took hours to restart safely, and postmortems began asking how long a from-zero restart takes and when it was last tested.
Practice and tooling shifts
- Cell-based architecture and shuffle sharding entered common vocabulary through AWS’s builders’ library and the Kinesis writeup.
- Incident tooling became its own product category (incident.io, FireHydrant, Rootly, Blameless), with Slack-native channels and auto-generated timelines.
- SLO platforms like Nobl9 turned error budgets into a product.
Takeaways that still hold
- List your transitive dependencies on us-east-1, or any single region, including your vendors’ and your status page’s.
- Design so the data plane survives the control plane dying. Test it by turning the control plane off.
- Treat OS limits (threads, file descriptors, conntrack) as capacity dimensions with alarms, not surprises.
- One customer’s config should never crash the fleet. Validate, sandbox, and canary customer-supplied input like code.