http://azarudeen.com/tags/automation/Recent content in Automation on Azarudeen.comHugoen-usMon, 01 Jan 2018 10:00:00 +0530http://azarudeen.com/posts/05-jan-2018-to-mar-2019-when-automation-fights-back/Mon, 01 Jan 2018 10:00:00 +0530http://azarudeen.com/posts/05-jan-2018-to-mar-2019-when-automation-fights-back/<h1 id="when-automation-fights-back-jan-2018--mar-2019">When Automation Fights Back (Jan 2018 – Mar 2019)</h1> <p>By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from &ldquo;why did the component fail?&rdquo; to &ldquo;why did our self-healing make it worse?&rdquo;</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>TSB Bank migration, April 2018</strong> — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain.</li> <li><strong>GitHub, October 21, 2018</strong> — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. <strong>Split-brain.</strong> GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (<a href="https://github.blog/2018-10-30-oct21-post-incident-analysis/">github.blog</a>).</li> <li><strong>Microsoft Azure South Central US, September 2018</strong> — A <strong>lightning strike</strong> caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many &ldquo;global&rdquo; Azure services (including Azure AD and the status portal) had hidden dependencies on one region.</li> <li><strong>Google Cloud, July 2018</strong> — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius.</li> <li><strong>Facebook, March 13, 2019</strong> — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a <strong>server configuration change</strong> — at the time the longest outage in the company&rsquo;s history.</li> <li><strong>Wells Fargo, February 2019</strong> — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. Automated failover needs a theory of data.</strong> GitHub&rsquo;s incident became <em>the</em> case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking &ldquo;what does our orchestrator do during a partition?&rdquo; — a Jepsen-style question applied to ops tooling.</p>