Shared Fate: Heartbleed, Mass Reboots, and the Limits of Cloud Trust
Shared Fate in the Cloud (Apr 2014 – Jun 2015) This window is when the industry learned that moving to the cloud means sharing your provider’s fate — their hypervisor patches, their config rollouts, and their operators’ keystrokes. It’s also when security incidents started being written up with the same discipline as availability incidents. The incidents that defined the period Heartbleed, April 2014 — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so “patch and rotate” took weeks. Shellshock (September 2014) repeated the drill for bash. Joyent, May 2014 — An operator running a routine update rebooted an entire data center’s worth of customer systems with one command. Joyent’s postmortem was admirably direct: the problem wasn’t the operator, it was that the tooling allowed a datacenter-wide target with no confirmation. A textbook blameless writeup. AWS Xen reboot, September 2014 — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the “design for instance failure” gospel (Netflix, famously) sailed through; those who hadn’t discovered pet servers the hard way. Microsoft Azure Storage, November 2014 — A performance fix was rolled out globally, skipping the staged “flighting” process, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever. NYSE, United Airlines, and the WSJ — July 8, 2015 — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles. What the postmortems reveal 1. Configuration change became the leading villain. The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the rollout of a config flag was the failure. “All deploys are staged, no exceptions, including configuration” started appearing in action items. ...