One Regex and a Pandemic (Apr 2019 – Jun 2020)
Filed July 2, 2020 — one year after the 27-minute regex outage this window is named for.
This window has a clean shape. It opens with self-inflicted global outages at Cloudflare and Google that pushed the industry toward staged rollouts, and it closes with COVID-19 stress-testing every capacity plan on the planet.
The incidents that defined the period
- Google Cloud, June 2, 2019. A maintenance automation event descheduled network control-plane jobs across several regions, and the resulting congestion throttled Google Cloud, YouTube, and Gmail for about four hours. The detail everyone remembers: the outage broke the very tools engineers needed to fix it.
- Cloudflare, July 2, 2019. A single WAF rule with a badly backtracking regex went out globally (WAF rules skipped staged rollout on purpose, for fast response to attacks) and pinned every CPU on Cloudflare’s edge. Twenty-seven minutes of global 502s, and one of the best postmortems ever written (blog.cloudflare.com), including a short lecture on regex complexity and why the kill switch was slow.
- Verizon BGP route leak, June 24, 2019. A small ISP’s route optimizer leaked routes through Verizon and blackholed chunks of the internet, Cloudflare included. Cloudflare’s blunt writeup (it called the incident “a small heart attack”) pushed RPKI adoption into the mainstream.
- Stripe, July 2019. Two linked database failures. Stripe published a detailed root-cause report, which was notable for a payments company.
- Salesforce, May 2019. A database script granted broad permissions across orgs, and the fix (revoking those permissions widely) caused more disruption than the bug. Recovery-as-a-second-incident got a name.
- COVID-19 surge, March–June 2020. Zoom grew about 30x. Robinhood had repeated trading-day outages from launch-day architecture under thundering-herd load. Streaming services in Europe voluntarily dropped quality, and unemployment systems on mainframes buckled. It was less one incident than a global load test.
What the postmortems reveal
Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose, for speed against attackers. The lesson wasn’t “never ship fast,” it was “your fastest pipeline needs the strongest safety switches.” Anything that goes global instantly became a flag in design reviews.
Plan to fix the system without the system. Google’s engineers were slowed by the same congestion they were debugging, and Cloudflare’s responders hit slowed internal access. Out-of-band access, break-glass credentials, and “emergency tools must not depend on production” became standard action items.
Capacity planning turned into scenario planning. COVID killed the assumption that load grows smoothly. Postmortems and readiness reviews started modeling sudden jumps: what if traffic triples on Monday?
Remote incident response worked. On-call teams scattered by necessity proved that incident command runs fine over video and chat, and that permanently changed assumptions about co-located war rooms.
Practice and tooling shifts
- Progressive delivery (feature flags, canaries, percentage rollouts, tools like LaunchDarkly) became the default answer to “how do we prevent the next bad global config push?”
- RPKI and BGP hygiene moved from IETF talks to deployment dashboards.
- Load shedding and graceful degradation got real engineering attention: serve stale data, drop non-critical features, queue writes.
Takeaways that still hold
- Every bypass of your staged rollout is a standing invitation to a global outage. Instrument and budget those bypasses.
- Keep a path to production that doesn’t go through production.
- Regexes are code. Lint them for backtracking, and put CPU budgets on rule engines.
- Your capacity plan needs a pandemic column: 3–30x, arriving in days.