Devops

Shared Fate in the Cloud (Apr 2014 – Jun 2015) This window is when the industry learned that moving to the cloud means sharing your provider’s fate — their hypervisor patches, their config rollouts, and their operators’ keystrokes. It’s also when security incidents started being written up with the same discipline as availability incidents. The incidents that defined the period Heartbleed, April 2014 — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so “patch and rotate” took weeks. Shellshock (September 2014) repeated the drill for bash. Joyent, May 2014 — An operator running a routine update rebooted an entire data center’s worth of customer systems with one command. Joyent’s postmortem was admirably direct: the problem wasn’t the operator, it was that the tooling allowed a datacenter-wide target with no confirmation. A textbook blameless writeup. AWS Xen reboot, September 2014 — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the “design for instance failure” gospel (Netflix, famously) sailed through; those who hadn’t discovered pet servers the hard way. Microsoft Azure Storage, November 2014 — A performance fix was rolled out globally, skipping the staged “flighting” process, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever. NYSE, United Airlines, and the WSJ — July 8, 2015 — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles. What the postmortems reveal 1. Configuration change became the leading villain. The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the rollout of a config flag was the failure. “All deploys are staged, no exceptions, including configuration” started appearing in action items. ...

The Blameless Awakening (Jan 2013 – Mar 2014) In early 2013, the public postmortem was still a novelty. Most companies treated outages as PR problems to be minimized, not learning opportunities to be shared. By the spring of 2014, that had visibly changed — and this 15-month window is where the modern postmortem culture took root. The incidents that defined the period Microsoft Azure, February 2013 — A worldwide Azure Storage outage caused by an expired SSL certificate. The lesson — certificate lifecycle management is an operational discipline, not a checkbox — still gets relearned every year. Google, August 2013 — Google went dark for roughly 2–5 minutes, and global internet traffic reportedly dropped ~40%. The first mainstream glimpse of how concentrated the web had become. Amazon.com, August 2013 — A ~30-minute outage of the retail site, widely used to popularize “downtime costs $X per minute” math in reliability business cases. NASDAQ “Flash Freeze,” August 2013 — A three-hour trading halt traced to a software flaw in the Securities Information Processor, showing that finance’s bespoke infrastructure had the same failure modes as web systems. HealthCare.gov, October 2013 — Not a cloud outage but the era’s defining systems failure: a big-bang launch with no load testing, no incremental rollout, and no operational ownership. Its rescue by a small team of web-industry engineers seeded what later became the US Digital Service — and became the canonical argument for DevOps practices in government and enterprise. What the postmortems reveal 1. “Blameless” went from Etsy blog post to industry norm. John Allspaw’s writing on blameless postmortems and Etsy’s open-sourced Morgue tool (their internal postmortem tracker) gave teams both the philosophy and the software. The core idea — engineers closest to the failure have the most information, and punishing them destroys that information — started appearing in conference talks everywhere. ...