Shared Fate in the Cloud (Apr 2014 – Jun 2015)
Filed July 12, 2015 — the weekend after the NYSE, United Airlines, and the WSJ all fell over on the same Wednesday.
This is when everyone learned that moving to the cloud means sharing your provider’s mistakes too: their patches, their config pushes, their operators' typos. Security incidents also started getting the same careful postmortem treatment as plain downtime.
The incidents that defined the period
- Heartbleed, April 2014. The OpenSSL bug that forced mass certificate rotation across the internet. The real operational lesson was that almost nobody had a list of where TLS actually terminated, so “patch and rotate” took weeks. Shellshock in September repeated the drill for bash.
- Joyent, May 2014. An operator running a routine update rebooted an entire data center of customer systems with one command. Joyent’s postmortem was direct: the problem wasn’t the operator, it was that the tooling let you target a whole datacenter with no confirmation.
- AWS Xen reboot, September 2014. AWS rebooted a large share of EC2 instances to patch a Xen bug before it was disclosed. Customers who had designed for instance failure barely noticed. Those who hadn’t found out the hard way which servers they’d been treating as pets.
- Microsoft Azure Storage, November 2014. A performance fix went out globally, skipping the usual staged rollout, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the team had gone around its own rollout policy. It’s one of the most-cited config-change postmortems there is.
- NYSE, United Airlines, and the WSJ, July 8, 2015. Three unrelated same-day outages that everyone assumed were connected. A good example of how reliability failures turn into a news cycle.
What the postmortems reveal
Configuration changes became the leading cause. The Azure writeup captured a pattern that still dominates postmortems: the code was fine, and the rollout of a config flag was what broke things. “All deploys are staged, including config” started showing up in action items.
Guardrails in tooling beat asking people to be careful. Joyent’s incident reframed human error. If one command can reboot a datacenter, the command is the bug. Blast-radius limits, confirmation prompts, and dry-run modes became standard follow-ups.
Security response started borrowing incident-response mechanics. Heartbleed and Shellshock were run like outages, with incident commanders, status pages, and rolling updates. The wall between “security incident” and “operational incident” started coming down.
Practice and tooling shifts
- Docker took off, and with it the first postmortems that treated container registries and image pipelines as critical infrastructure.
- Kubernetes 1.0 shipped in mid-2015, right at the end of this window. It would go on to cause about half the postmortems of the next decade.
- Game days spread past Amazon and Netflix. Teams started rehearsing failovers instead of only documenting them.
Takeaways that still hold
- Your provider’s maintenance is your incident. Design so reboots are boring.
- Staged-rollout policies only work if the tooling makes skipping them harder than following them.
- Keep a list of everywhere a risky dependency or certificate lives. The patch is quick; finding all the places is slow.