One Regex and a Pandemic (Apr 2019 – Jun 2020)
This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth.
The incidents that defined the period
- Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage.
- Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow.
- Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream.
- Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company.
- Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern.
- COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test.
What the postmortems reveal
1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews.
2. Plan for fixing the system without the system. Google’s engineers were hampered by the very congestion they were debugging; Cloudflare’s responders hit slowed internal access. Out-of-band access, break-glass credentials, and “emergency tooling must not depend on production” became standard action items.
3. Capacity planning became scenario planning. COVID demolished the assumption that load grows smoothly. Postmortems and readiness reviews began modeling step-function demand: what if traffic triples on Monday?
4. Remote incident response worked. Distributed-by-necessity on-call proved that incident command runs fine over video and chat — permanently changing assumptions about co-located war rooms.
Practice and tooling shifts
- Progressive delivery (feature flags, canaries, percentage rollouts — LaunchDarkly et al.) became the default answer to “how do we prevent the next global config push?”
- RPKI and BGP hygiene moved from IETF talk to deployment dashboards.
- Load shedding and graceful degradation got first-class engineering attention (serve stale, drop noncritical features, queue writes).
Takeaways that still hold
- Every bypass of your staged rollout is a standing invitation to a global outage; instrument and budget those bypasses.
- Keep a path to production that doesn’t traverse production.
- Regexes are code; lint them for backtracking, and put CPU budgets on rule engines.
- Your capacity plan needs a pandemic column: 3–30x, arriving in days.