http://azarudeen.com/tags/configuration/Recent content in Configuration on Azarudeen.comHugoen-usTue, 01 Apr 2014 10:00:00 +0530http://azarudeen.com/posts/02-apr-2014-to-jun-2015-shared-fate-in-the-cloud/Tue, 01 Apr 2014 10:00:00 +0530http://azarudeen.com/posts/02-apr-2014-to-jun-2015-shared-fate-in-the-cloud/<h1 id="shared-fate-in-the-cloud-apr-2014--jun-2015">Shared Fate in the Cloud (Apr 2014 – Jun 2015)</h1> <p>This window is when the industry learned that moving to the cloud means sharing your provider&rsquo;s fate — their hypervisor patches, their config rollouts, and their operators&rsquo; keystrokes. It&rsquo;s also when security incidents started being written up with the same discipline as availability incidents.</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>Heartbleed, April 2014</strong> — The OpenSSL bug that forced mass certificate rotation across the internet. Its real operational lesson: almost nobody had an inventory of where TLS terminated, so &ldquo;patch and rotate&rdquo; took weeks. Shellshock (September 2014) repeated the drill for bash.</li> <li><strong>Joyent, May 2014</strong> — An operator running a routine update <strong>rebooted an entire data center&rsquo;s worth of customer systems</strong> with one command. Joyent&rsquo;s postmortem was admirably direct: the problem wasn&rsquo;t the operator, it was that the tooling <em>allowed</em> a datacenter-wide target with no confirmation. A textbook blameless writeup.</li> <li><strong>AWS Xen reboot, September 2014</strong> — AWS rebooted a large fraction of EC2 instances to patch a Xen vulnerability before disclosure. Customers who had followed the &ldquo;design for instance failure&rdquo; gospel (Netflix, famously) sailed through; those who hadn&rsquo;t discovered pet servers the hard way.</li> <li><strong>Microsoft Azure Storage, November 2014</strong> — A performance fix was rolled out <strong>globally, skipping the staged &ldquo;flighting&rdquo; process</strong>, and an infinite loop in the blob frontends took down storage across regions. Microsoft&rsquo;s postmortem admitted the human deviation from their own rollout policy — one of the most cited config-change postmortems ever.</li> <li><strong>NYSE, United Airlines, and the WSJ — July 8, 2015</strong> — Three unrelated same-day outages that the public assumed were connected. A lesson in how reliability failures become news cycles.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. Configuration change became the leading villain.</strong> The Azure writeup crystallized a pattern that dominates postmortems to this day: the code was fine; the <em>rollout</em> of a config flag was the failure. &ldquo;All deploys are staged, no exceptions, including configuration&rdquo; started appearing in action items.</p>