The SRE Book Era (Jul 2015 – Sep 2016)

Google published Site Reliability Engineering in April 2016 and handed the industry a shared vocabulary: SLOs, error budgets, toil, and a whole chapter on postmortem culture. Meanwhile, the period’s biggest incidents were masterclasses in cascading failure — systems that fell over not from the initial fault, but from their own recovery behavior.

The incidents that defined the period

  • AWS DynamoDB, September 20, 2015 — The canonical cascading-failure postmortem (aws.amazon.com/message/5467D2). A network disruption caused storage servers to re-request membership metadata simultaneously; the metadata service, already near capacity from a new index feature, couldn’t serve the herd; retries made it worse. DynamoDB’s outage cascaded into EC2, SQS, and CloudWatch in us-east-1. Action items — capacity headroom, longer timeouts, segmented retries — read like a distributed-systems syllabus.
  • Salesforce NA14, May 2016 — A database failure plus a failed failover left a major instance degraded for nearly a day, with some data unrecoverable. It pushed “your SaaS vendor’s DR plan is your DR plan” into procurement conversations.
  • Southwest Airlines (July 2016) and Delta (August 2016) — Back-to-back airline meltdowns from single-point-of-failure infrastructure (a failed router; a datacenter power incident) cancelling thousands of flights. Boards started asking about technical debt.
  • Telstra, 2016 — A string of national mobile outages in Australia, one triggered by a single node being taken offline incorrectly, normalized the telco postmortem press release.

What the postmortems reveal

1. Retry storms became a named enemy. The DynamoDB writeup made “metastable failure” patterns mainstream years before the academic term: exponential backoff, jitter, circuit breakers, and load shedding moved from Netflix blog posts into default library behavior (and into everyone’s action items).

2. Postmortems got a template. The SRE book’s postmortem chapter — impact, root causes, trigger, detection, action items with owners, lessons learned, timeline — became the format. Google publishing its internal template did for incident writeups what RFC formats did for protocols.

3. Error budgets reframed the reliability argument. Instead of “ops wants stability, devs want velocity,” teams could say: we have budget, ship; we’re out of budget, harden. Postmortems began referencing SLO burn instead of raw downtime minutes.

Practice and tooling shifts

  • SREcon grew into the industry’s postmortem exchange, with talks that were essentially public incident reviews.
  • Kubernetes and microservices adoption accelerated, moving failure modes from “the server died” to “the retry policy between services is wrong.”
  • Status pages professionalized (StatusPage.io acquired by Atlassian, 2016) — customer communication became part of the incident, not an afterthought.

Takeaways that still hold

  1. The initial fault is rarely what takes you down; your system’s reaction to the fault is. Test recovery behavior under load, not just failure.
  2. Capacity planning must include the recovery herd: what happens when every client reconnects at once?
  3. Adopt a postmortem template. The discipline of filling in “detection” and “action items with owners” is worth more than any individual finding.