The Blameless Awakening (Jan 2013 – Mar 2014)

Filed April 3, 2014 — the week healthcare.gov closed its first open enrollment, ending the saga that taught the industry to read postmortems.

In early 2013, few companies published real postmortems. Outages were treated as PR problems, not things worth explaining. That changed fast. By 2014, writing an honest postmortem was becoming normal, and this 15-month window is where the shift started.

The incidents that defined the period

  • Microsoft Azure, February 2013. An expired SSL certificate took down Azure Storage worldwide. The lesson that certificates need active tracking, not a one-time setup, still gets relearned every year.
  • Google, August 2013. Google went dark for a few minutes and global internet traffic reportedly dropped around 40%. It was the first time many people saw how much of the web ran through one company.
  • Amazon.com, August 2013. A roughly 30-minute outage of the retail site. This is the one people cite when they do the “downtime costs $X per minute” math.
  • NASDAQ Flash Freeze, August 2013. A software flaw in the Securities Information Processor halted trading for three hours. Finance’s custom systems turned out to fail the same way web systems do.
  • HealthCare.gov, October 2013. Not a cloud outage, but the defining systems failure of the era: a big launch with no load testing, no gradual rollout, and no clear operational owner. The small team that rescued it seeded what became the US Digital Service, and it became the standard argument for DevOps in government.

What the postmortems reveal

Blameless postmortems went from an Etsy blog post to an industry norm. John Allspaw’s writing and Etsy’s open-sourced Morgue tool gave teams both the idea and the software. The core point is simple: the engineers closest to a failure know the most about it, and punishing them just makes people stop talking.

Root cause stopped being a single thing. The better postmortems dropped “the root cause was X” and started listing contributing factors instead: an expired cert, plus no monitoring for it, plus a deploy process that couldn’t roll back. Richard Cook’s How Complex Systems Fail got passed around a lot.

Chaos engineering left the lab. Netflix had open-sourced Chaos Monkey in 2012, and their writeups about surviving AWS instance and zone failures were basically postmortems written before the outage.

Practice and tooling shifts

  • ChatOps caught on. GitHub’s Hubot made it normal to run deploys and checks from chat, which meant incident timelines started recording themselves.
  • On-call became a product. PagerDuty and VictorOps turned paging from homegrown cron scripts into real software.
  • Smaller, more frequent deploys started showing up in postmortems as the fix. Big-bang releases kept appearing as a contributing factor, with HealthCare.gov as the clearest example.

Takeaways that still hold

  1. Certificates, quotas, and licenses expire on a schedule. Your monitoring should know that schedule before the outage does.
  2. A postmortem that blames a person has failed. One that finds a missing guardrail has done its job.
  3. If you can’t roll out gradually, you’re choosing to find all your failure modes on the same day.