http://azarudeen.com/tags/cascading-failure/Recent content in Cascading-Failure on Azarudeen.comHugoen-usWed, 01 Jul 2015 10:00:00 +0530http://azarudeen.com/posts/03-jul-2015-to-sep-2016-the-sre-book-era/Wed, 01 Jul 2015 10:00:00 +0530http://azarudeen.com/posts/03-jul-2015-to-sep-2016-the-sre-book-era/<h1 id="the-sre-book-era-jul-2015--sep-2016">The SRE Book Era (Jul 2015 – Sep 2016)</h1> <p>Google published <em>Site Reliability Engineering</em> in April 2016 and handed the industry a shared vocabulary: SLOs, error budgets, toil, and a whole chapter on postmortem culture. Meanwhile, the period&rsquo;s biggest incidents were masterclasses in cascading failure — systems that fell over not from the initial fault, but from their own recovery behavior.</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>AWS DynamoDB, September 20, 2015</strong> — The canonical cascading-failure postmortem (<a href="https://aws.amazon.com/message/5467D2/">aws.amazon.com/message/5467D2</a>). A network disruption caused storage servers to re-request membership metadata simultaneously; the metadata service, already near capacity from a new index feature, couldn&rsquo;t serve the herd; retries made it worse. DynamoDB&rsquo;s outage cascaded into EC2, SQS, and CloudWatch in us-east-1. Action items — capacity headroom, longer timeouts, segmented retries — read like a distributed-systems syllabus.</li> <li><strong>Salesforce NA14, May 2016</strong> — A database failure plus a failed failover left a major instance degraded for nearly a day, with some data unrecoverable. It pushed &ldquo;your SaaS vendor&rsquo;s DR plan is your DR plan&rdquo; into procurement conversations.</li> <li><strong>Southwest Airlines (July 2016) and Delta (August 2016)</strong> — Back-to-back airline meltdowns from single-point-of-failure infrastructure (a failed router; a datacenter power incident) cancelling thousands of flights. Boards started asking about technical debt.</li> <li><strong>Telstra, 2016</strong> — A string of national mobile outages in Australia, one triggered by a single node being taken offline incorrectly, normalized the telco postmortem press release.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. Retry storms became a named enemy.</strong> The DynamoDB writeup made &ldquo;metastable failure&rdquo; patterns mainstream years before the academic term: exponential backoff, jitter, circuit breakers, and load shedding moved from Netflix blog posts into default library behavior (and into everyone&rsquo;s action items).</p>