http://azarudeen.com/tags/incident-response/Recent content in Incident-Response on Azarudeen.comHugoen-usTue, 01 Jan 2013 10:00:00 +0530http://azarudeen.com/posts/01-jan-2013-to-mar-2014-the-blameless-awakening/Tue, 01 Jan 2013 10:00:00 +0530http://azarudeen.com/posts/01-jan-2013-to-mar-2014-the-blameless-awakening/<h1 id="the-blameless-awakening-jan-2013--mar-2014">The Blameless Awakening (Jan 2013 – Mar 2014)</h1> <p>In early 2013, the public postmortem was still a novelty. Most companies treated outages as PR problems to be minimized, not learning opportunities to be shared. By the spring of 2014, that had visibly changed — and this 15-month window is where the modern postmortem culture took root.</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>Microsoft Azure, February 2013</strong> — A worldwide Azure Storage outage caused by an <strong>expired SSL certificate</strong>. The lesson — certificate lifecycle management is an operational discipline, not a checkbox — still gets relearned every year.</li> <li><strong>Google, August 2013</strong> — Google went dark for roughly 2–5 minutes, and global internet traffic reportedly dropped ~40%. The first mainstream glimpse of how concentrated the web had become.</li> <li><strong>Amazon.com, August 2013</strong> — A ~30-minute outage of the retail site, widely used to popularize &ldquo;downtime costs $X per minute&rdquo; math in reliability business cases.</li> <li><strong>NASDAQ &ldquo;Flash Freeze,&rdquo; August 2013</strong> — A three-hour trading halt traced to a software flaw in the Securities Information Processor, showing that finance&rsquo;s bespoke infrastructure had the same failure modes as web systems.</li> <li><strong>HealthCare.gov, October 2013</strong> — Not a cloud outage but the era&rsquo;s defining systems failure: a big-bang launch with no load testing, no incremental rollout, and no operational ownership. Its rescue by a small team of web-industry engineers seeded what later became the US Digital Service — and became the canonical argument for DevOps practices in government and enterprise.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. &ldquo;Blameless&rdquo; went from Etsy blog post to industry norm.</strong> John Allspaw&rsquo;s writing on blameless postmortems and Etsy&rsquo;s open-sourced <strong>Morgue</strong> tool (their internal postmortem tracker) gave teams both the philosophy and the software. The core idea — engineers closest to the failure have the most information, and punishing them destroys that information — started appearing in conference talks everywhere.</p>