The SRE Book Era (Jul 2015 – Sep 2016)
Filed September 20, 2016 — one year to the day after the DynamoDB cascade that defined this window.
Google published Site Reliability Engineering in 2016 and gave the industry a shared vocabulary: SLOs, error budgets, toil, blameless postmortems. At the same time, the era’s biggest outages kept showing the same pattern. Systems broke not from the original fault, but from how they tried to recover.
The incidents that defined the period
- AWS DynamoDB, September 20, 2015. The classic cascading-failure postmortem (aws.amazon.com/message/5467D2). A network blip made storage servers all re-request their membership data at once. The metadata service was already near capacity from a new feature and couldn’t keep up, and retries made it worse. The outage spread from DynamoDB into EC2, SQS, and CloudWatch in us-east-1. The fixes (more headroom, longer timeouts, smarter retries) read like a distributed-systems checklist.
- Salesforce NA14, May 2016. A database failure plus a failed failover left a major instance degraded for almost a day, with some data lost for good. It put “your SaaS vendor’s DR plan is your DR plan” into procurement conversations.
- Southwest (July 2016) and Delta (August 2016). Back-to-back airline meltdowns from single points of failure (a failed router, a datacenter power problem) cancelled thousands of flights. Boards started asking about technical debt.
- Telstra, 2016. A run of national mobile outages in Australia, one caused by taking a single node offline the wrong way, made the telco postmortem press release routine.
What the postmortems reveal
Retry storms got a name and a set of standard fixes. The DynamoDB writeup made this failure pattern mainstream well before the academic term “metastable failure” existed. Exponential backoff, jitter, circuit breakers, and load shedding moved out of Netflix blog posts and into default library behavior.
Postmortems got a template. The SRE book’s chapter (impact, root causes, trigger, detection, action items with owners, lessons, timeline) became the format. Google publishing its internal template did for incident writeups what RFCs did for protocols.
Error budgets changed the reliability argument. Instead of “ops wants stability, devs want speed,” a team could say: we have budget, so ship; we’re out of budget, so harden. Postmortems started citing SLO burn instead of raw downtime minutes.
Practice and tooling shifts
- SREcon grew into the industry’s place to trade postmortems, with talks that were really public incident reviews.
- Kubernetes and microservices adoption sped up, moving failures from “the server died” to “the retry policy between two services is wrong.”
- Status pages got professional. Atlassian bought StatusPage.io in 2016, and customer communication became part of the incident rather than an afterthought.
Takeaways that still hold
- The first fault rarely takes you down. How your system reacts to it does. Test recovery under load, not just the failure itself.
- Capacity planning has to include the recovery herd. What happens when every client reconnects at once?
- Use a postmortem template. Filling in “detection” and “action items with owners” is worth more than any single finding.