Sre on Azarudeen.com

The SRE Book Era: Error Budgets Meet Cascading Failure

Wed, 01 Jul 2015 10:00:00 +0530

The SRE Book Era (Jul 2015 – Sep 2016)

Google published Site Reliability Engineering in April 2016 and handed the industry a shared vocabulary: SLOs, error budgets, toil, and a whole chapter on postmortem culture. Meanwhile, the period’s biggest incidents were masterclasses in cascading failure — systems that fell over not from the initial fault, but from their own recovery behavior.

The incidents that defined the period

AWS DynamoDB, September 20, 2015 — The canonical cascading-failure postmortem (aws.amazon.com/message/5467D2). A network disruption caused storage servers to re-request membership metadata simultaneously; the metadata service, already near capacity from a new index feature, couldn’t serve the herd; retries made it worse. DynamoDB’s outage cascaded into EC2, SQS, and CloudWatch in us-east-1. Action items — capacity headroom, longer timeouts, segmented retries — read like a distributed-systems syllabus.
Salesforce NA14, May 2016 — A database failure plus a failed failover left a major instance degraded for nearly a day, with some data unrecoverable. It pushed “your SaaS vendor’s DR plan is your DR plan” into procurement conversations.
Southwest Airlines (July 2016) and Delta (August 2016) — Back-to-back airline meltdowns from single-point-of-failure infrastructure (a failed router; a datacenter power incident) cancelling thousands of flights. Boards started asking about technical debt.
Telstra, 2016 — A string of national mobile outages in Australia, one triggered by a single node being taken offline incorrectly, normalized the telco postmortem press release.

What the postmortems reveal

1. Retry storms became a named enemy. The DynamoDB writeup made “metastable failure” patterns mainstream years before the academic term: exponential backoff, jitter, circuit breakers, and load shedding moved from Netflix blog posts into default library behavior (and into everyone’s action items).

Shared Fate: Heartbleed, Mass Reboots, and the Limits of Cloud Trust

Tue, 01 Apr 2014 10:00:00 +0530

Shared Fate in the Cloud (Apr 2014 – Jun 2015)

This window is when the industry learned that moving to the cloud means sharing your provider’s fate — their hypervisor patches, their config rollouts, and their operators’ keystrokes. It’s also when security incidents started being written up with the same discipline as availability incidents.

The incidents that defined the period

Heartbleed, April 2014 — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so “patch and rotate” took weeks. Shellshock (September 2014) repeated the drill for bash.
Joyent, May 2014 — An operator running a routine update rebooted an entire data center’s worth of customer systems with one command. Joyent’s postmortem was admirably direct: the problem wasn’t the operator, it was that the tooling allowed a datacenter-wide target with no confirmation. A textbook blameless writeup.
AWS Xen reboot, September 2014 — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the “design for instance failure” gospel (Netflix, famously) sailed through; those who hadn’t discovered pet servers the hard way.
Microsoft Azure Storage, November 2014 — A performance fix was rolled out globally, skipping the staged “flighting” process, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever.
NYSE, United Airlines, and the WSJ — July 8, 2015 — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles.

What the postmortems reveal

1. Configuration change became the leading villain. The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the rollout of a config flag was the failure. “All deploys are staged, no exceptions, including configuration” started appearing in action items.

The Blameless Awakening: How Postmortems Became Engineering Culture

Tue, 01 Jan 2013 10:00:00 +0530

The Blameless Awakening (Jan 2013 – Mar 2014)

In early 2013, the public postmortem was still a novelty. Most companies treated outages as PR problems to be minimized, not learning opportunities to be shared. By the spring of 2014, that had visibly changed — and this 15-month window is where the modern postmortem culture took root.

The incidents that defined the period

Microsoft Azure, February 2013 — A worldwide Azure Storage outage caused by an expired SSL certificate. The lesson — certificate lifecycle management is an operational discipline, not a checkbox — still gets relearned every year.
Google, August 2013 — Google went dark for roughly 2–5 minutes, and global internet traffic reportedly dropped ~40%. The first mainstream glimpse of how concentrated the web had become.
Amazon.com, August 2013 — A ~30-minute outage of the retail site, widely used to popularize “downtime costs $X per minute” math in reliability business cases.
NASDAQ “Flash Freeze,” August 2013 — A three-hour trading halt traced to a software flaw in the Securities Information Processor, showing that finance’s bespoke infrastructure had the same failure modes as web systems.
HealthCare.gov, October 2013 — Not a cloud outage but the era’s defining systems failure: a big-bang launch with no load testing, no incremental rollout, and no operational ownership. Its rescue by a small team of web-industry engineers seeded what later became the US Digital Service — and became the canonical argument for DevOps practices in government and enterprise.

What the postmortems reveal

1. “Blameless” went from Etsy blog post to industry norm. John Allspaw’s writing on blameless postmortems and Etsy’s open-sourced Morgue tool (their internal postmortem tracker) gave teams both the philosophy and the software. The core idea — engineers closest to the failure have the most information, and punishing them destroys that information — started appearing in conference talks everywhere.