Shared Fate in the Cloud (Apr 2014 – Jun 2015)
This window is when the industry learned that moving to the cloud means sharing your provider’s fate — their hypervisor patches, their config rollouts, and their operators’ keystrokes. It’s also when security incidents started being written up with the same discipline as availability incidents.
The incidents that defined the period
- Heartbleed, April 2014 — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so “patch and rotate” took weeks. Shellshock (September 2014) repeated the drill for bash.
- Joyent, May 2014 — An operator running a routine update rebooted an entire data center’s worth of customer systems with one command. Joyent’s postmortem was admirably direct: the problem wasn’t the operator, it was that the tooling allowed a datacenter-wide target with no confirmation. A textbook blameless writeup.
- AWS Xen reboot, September 2014 — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the “design for instance failure” gospel (Netflix, famously) sailed through; those who hadn’t discovered pet servers the hard way.
- Microsoft Azure Storage, November 2014 — A performance fix was rolled out globally, skipping the staged “flighting” process, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever.
- NYSE, United Airlines, and the WSJ — July 8, 2015 — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles.
What the postmortems reveal
1. Configuration change became the leading villain. The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the rollout of a config flag was the failure. “All deploys are staged, no exceptions, including configuration” started appearing in action items.
2. Tooling guardrails over operator discipline. Joyent’s incident reframed human error: if one command can reboot a datacenter, the command is the bug. Blast-radius limits, confirmation prompts, and dry-run modes became standard action items.
3. Security response adopted incident response mechanics. Heartbleed and Shellshock were run like outages — incident commanders, status pages, rolling comms. The wall between “security incident” and “operational incident” started coming down.
Practice and tooling shifts
- Docker mania began, and with it the first postmortems involving container registries and image pipelines as critical infrastructure.
- Kubernetes 1.0 shipped (June/July 2015) at the very end of this window — the source of roughly half of all postmortems for the following decade.
- Game days spread beyond Amazon and Netflix: teams began rehearsing failovers instead of just documenting them.
Takeaways that still hold
- Your provider’s maintenance is your incident; architect so reboots are boring.
- Staged rollout policies only work if tooling makes skipping them harder than following them.
- Keep an inventory of every place a vulnerable dependency (or certificate) lives — the patch is fast, the finding is slow.