Google-Cloud

The CrowdStrike Reckoning (Apr 2024 – Jun 2025) One Friday in July 2024 produced the largest IT outage in history — and it wasn’t a cloud provider. This window’s postmortems are dominated by other people’s software running inside your trust boundary: security agents in the kernel, a dealer platform for an entire industry, a cloud vendor deleting a customer, and a quota policy pushed worldwide. The incidents that defined the period Google Cloud / UniSuper, May 2024 — A misconfiguration during provisioning led Google Cloud to delete an entire customer’s private cloud subscription — a ~$125B pension fund — causing ~two weeks of disruption. Recovery leaned on UniSuper’s own third-party backups. The joint apology statement was unprecedented; “what if our cloud account itself is the failure domain?” entered every DR review. CDK Global, June 2024 — Ransomware took down the SaaS platform underpinning ~15,000 North American car dealerships for weeks. A whole industry discovered it had a single point of failure it had never load-tested: its vendor. CrowdStrike, July 19, 2024 — A faulty Rapid Response Content update (Channel File 291) hit an out-of-bounds read in the Falcon sensor running in the Windows kernel: ~8.5 million machines blue-screened. Airlines, hospitals, banks, 911 centers. Insured losses in the billions; Delta alone claimed ~$500M. The RCA and Congressional testimony detailed the gap: sensor code was staged and tested; content updates were validated by a checker with a bug and deployed globally at once (crowdstrike.com RCA). Azure Central US, July 30, 2024 — A DDoS defense misconfiguration amplified rather than mitigated an attack, in a summer of repeated Microsoft incidents. OpenAI, December 11, 2024 — A new telemetry service overwhelmed Kubernetes API servers across clusters; DNS caching masked the rollout risk, and engineers were locked out of the control planes they needed to revert. A modern classic: observability tooling as the outage trigger, published with unusual candor for an AI lab. Google Cloud, June 12, 2025 — A new Service Control policy feature with a null-pointer path, no feature flag, and instant global metadata replication crash-looped API management worldwide (~3 hours; ~7.5h for us-central1). Cloudflare (whose Workers KV depended on GCS), Spotify, and dozens of others went down with it. The postmortem’s own action items read like this series' greatest hits: flag-gate everything, stagger global propagation, add backoff. What the postmortems reveal 1. “Content” updates are code. CrowdStrike’s split — rigorous staging for binaries, instant global push for configuration content — is the same pattern as Cloudflare 2019 and Google 2025. The industry’s hardest-won lesson keeps recurring one abstraction level up: anything that changes runtime behavior needs canaries, whether it’s called code, config, content, or policy. ...

One Regex and a Pandemic (Apr 2019 – Jun 2020) This window bookends neatly: it opens with self-inflicted global outages at Cloudflare and Google that sharpened the industry’s thinking about staged rollouts, and closes with COVID-19 stress-testing every capacity plan on Earth. The incidents that defined the period Google Cloud, June 2, 2019 — A maintenance automation event descheduled network control-plane jobs across multiple regions; congestion throttled Google Cloud, YouTube, and Gmail for ~4 hours. The postmortem detail everyone remembers: the outage impaired the tools engineers needed to fix the outage. Cloudflare, July 2, 2019 — A single WAF rule containing a catastrophically backtracking regex was pushed globally (WAF rules were exempt from staged rollout, by design, for emergency response) and pinned every CPU on Cloudflare’s edge. 27 minutes of global 502s, and one of the finest postmortems ever written (blog.cloudflare.com) — including a mini-lecture on regex complexity and why their kill switch was slow. Verizon BGP route leak, June 24, 2019 — A small ISP’s route optimizer leaked routes through Verizon, blackholing chunks of the internet including Cloudflare. Cloudflare’s blunt public writeup (“a small heart attack”) pushed RPKI adoption into the mainstream. Stripe, July 2019 — Two coupled database failures; Stripe published a detailed root-cause report, notable for a payments company. Salesforce, May 2019 — A database script granted broad permissions across orgs; the remediation (revoking permissions widely) caused more disruption than the bug. Recovery-as-second-incident became a named pattern. COVID-19 surge, March–June 2020 — Zoom grew ~30x; Robinhood suffered repeated trading-day outages (thundering-herd load on launch-day architecture); streaming services voluntarily degraded quality in Europe; unemployment systems running on mainframes buckled. Not one incident but a planetary load test. What the postmortems reveal 1. Emergency paths are the most dangerous paths. Cloudflare’s WAF pipeline skipped staged rollout on purpose — speed against attackers. The lesson wasn’t “never ship fast” but “your fastest pipeline needs the strongest circuit breakers.” Global-instant anything became a red flag in design reviews. ...