http://azarudeen.com/tags/capacity/Recent content in Capacity on Azarudeen.comHugoen-usMon, 01 Apr 2019 10:00:00 +0530http://azarudeen.com/posts/06-apr-2019-to-jun-2020-one-regex-and-a-pandemic/Mon, 01 Apr 2019 10:00:00 +0530http://azarudeen.com/posts/06-apr-2019-to-jun-2020-one-regex-and-a-pandemic/<h1 id="one-regex-and-a-pandemic-apr-2019--jun-2020">One Regex and a Pandemic (Apr 2019 – Jun 2020)</h1> <p>This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry&rsquo;s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth.</p> <h2 id="the-incidents-that-defined-the-period">The incidents that defined the period</h2> <ul> <li><strong>Google Cloud, June 2, 2019</strong> — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: <strong>the outage impaired the tools engineers needed to fix the outage.</strong></li> <li><strong>Cloudflare, July 2, 2019</strong> — A single WAF rule containing a <strong>catastrophically backtracking regex</strong> was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare&rsquo;s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (<a href="https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/">blog.cloudflare.com</a>) — including a mini-lecture on regex complexity and why their kill switch was slow.</li> <li><strong>Verizon BGP route leak, June 24, 2019</strong> — A small ISP&rsquo;s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare&rsquo;s blunt public writeup (&ldquo;a small heart attack&rdquo;) pushed RPKI adoption into the mainstream.</li> <li><strong>Stripe, July 2019</strong> — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company.</li> <li><strong>Salesforce, May 2019</strong> — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern.</li> <li><strong>COVID-19 surge, March–June 2020</strong> — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test.</li> </ul> <h2 id="what-the-postmortems-reveal">What the postmortems reveal</h2> <p><strong>1. Emergency paths are the most dangerous paths.</strong> Cloudflare&rsquo;s WAF pipeline skipped staged rollout <em>on purpose</em> — speed against attackers. The lesson wasn&rsquo;t &ldquo;never ship fast&rdquo; but &ldquo;your fastest pipeline needs the strongest circuit breakers.&rdquo; Global-instant anything became a red flag in design reviews.</p>