Aws

Agents On Call (Jul 2025 – Aug 2026) This window opened with a dormant config error and got worse from there. AWS, Azure, and Cloudflare each had a headline global outage within a single month of autumn 2025, and “the internet is three companies in a trench coat” became a mainstream joke. A year later, July 2026 delivered four provider incidents in nine days. Underneath all of it, the biggest shift since the SRE book has been quietly happening: AI agents moving from summarizing incidents to actually responding to them. ...

When the Map Burns with the Territory (Oct 2021 – Dec 2022) Filed January 11, 2023 — finished, fittingly, on the morning the FAA’s NOTAM outage grounded US departures and opened the next chapter. The image that defines this window: Facebook engineers reportedly couldn’t badge into their own buildings, because the outage had also taken down the door systems. Again and again, the tools meant to fix an outage turned out to depend on the thing that was broken. ...

The us-east-1 Problem (Jul 2020 – Sep 2021) Filed November 25, 2021 — the first anniversary of the Kinesis event, the archetypal us-east-1 story. The incidents in this window share a shape. Something small and deep (a thread limit, a quota system, one customer’s config change) spreads out until half the internet feels it. Readers started asking a new first question: what does everything else quietly depend on? The incidents that defined the period AWS Kinesis / us-east-1, November 25, 2020. Adding capacity to Kinesis’s front-end fleet pushed servers past an OS thread limit. The fleet needed a slow full restart, and services that depended on it (Cognito, CloudWatch, and various vendors’ status pages) went down with it (aws.amazon.com/message/11201). The postmortem taught a lot of engineers what a cell-based architecture is by describing what it looks like to not have one. Google, December 14, 2020. The identity and quota system took down Gmail, YouTube, and Google Cloud auth for about 47 minutes. An automated quota migration reported usage as zero and starved the auth service. Safety systems that can’t tell “no usage” from “no data” became a recurring theme. Slack, January 4, 2021. First workday of the year. Provisioning couldn’t scale up in AWS fast enough, and Slack’s own dashboards were degraded during the response (slack.engineering). OVHcloud fire, March 2021. A Strasbourg datacenter burned, and some customers learned their backups were in the building that was on fire. Physical DR came back into the conversation. Fastly, June 8, 2021. A dormant bug shipped in May was triggered by one customer’s valid config change and dropped about 85% of Fastly’s network. Global outage in seconds, found in minutes, mostly restored within an hour (fastly.com). Reuters, gov.uk, and Amazon went dark together. Forty-nine minutes that made “CDN concentration” a mainstream news topic. Akamai Edge DNS, July 2021. A bug triggered by a config update took down banks and airlines for about an hour. Same lesson, different CDN. What the postmortems reveal Control plane versus data plane became the sharpest way to look at these. Google’s quota system, AWS’s front-end metadata fleet, Fastly’s config distribution: in each case the management machinery failed while the actual capacity was fine. “Static stability,” meaning the data plane keeps working when the control plane is down, became the goal to design toward. ...

Typos That Broke the Internet (Oct 2016 – Dec 2017) Filed February 4, 2018 — the weekend after GitLab’s live-streamed database recovery turned one. This is the window that proved honest postmortems build trust instead of costing it. A livestreamed database recovery and a single typo that broke half the web produced two of the most-read incident reports ever written. The incidents that defined the period Dyn DNS DDoS, October 21, 2016. The Mirai botnet, built from hacked IoT devices, took down a big managed-DNS provider and, with it, Twitter, Netflix, Reddit, and GitHub for most of a day. It was the industry’s introduction to dependency concentration: dozens of “independent” sites all used one DNS provider. GitLab, January 31, 2017. An exhausted engineer fighting replication lag ran rm -rf on the primary’s data directory. Five different backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute writeup (about.gitlab.com). About six hours of data was lost, and GitLab’s reputation arguably went up. AWS S3 us-east-1, February 28, 2017. An operator debugging the billing system mistyped a command and removed far more capacity than intended. The index subsystem needed a full restart it hadn’t done in years (aws.amazon.com/message/41926). Thousands of sites broke, including AWS’s own status page, whose health icons were hosted on S3. Cloudbleed, February 2017. A parser bug leaked memory from one Cloudflare customer into other customers’ cached pages. Cloudflare’s detailed disclosure set a new bar for security postmortems. British Airways, May 2017. A datacenter power event grounded flights worldwide. The vague public explanation became the counterexample to GitLab-style openness. Equifax breach, 2017. An unpatched Struts vulnerability. The lesson was less about the bug and more about knowing what you run and keeping it patched. What the postmortems reveal Transparency won, clearly. GitLab and AWS gave specifics: the command, the parameter, the checks they were adding. BA gave vagueness. Customers noticed which companies they trusted more afterward, and “publish the real postmortem” became a strategy rather than a legal risk. ...