The CrowdStrike Reckoning (Apr 2024 – Jun 2025)

Filed July 19, 2025 — one year to the day after Channel File 291.

One Friday in July 2024 caused the largest IT outage in history, and it wasn’t a cloud provider. This window’s postmortems keep circling back to the same theme: other people’s software running inside your trust boundary. A security agent in the kernel, a dealer platform an entire industry depends on, a cloud vendor deleting a customer’s data, a quota policy pushed everywhere at once.

The incidents that defined the period

  • Google Cloud / UniSuper, May 2024. A misconfiguration during provisioning led Google Cloud to delete an entire customer’s private cloud subscription, a roughly $125B pension fund, causing about two weeks of disruption. Recovery leaned on UniSuper’s own third-party backups. The joint apology was unprecedented, and “what if our cloud account itself is the failure domain?” entered every DR review.
  • CDK Global, June 2024. Ransomware took down the SaaS platform behind about 15,000 North American car dealerships for weeks. A whole industry found out it had a single point of failure it had never load-tested: its vendor.
  • CrowdStrike, July 19, 2024. A faulty Rapid Response Content update (Channel File 291) hit an out-of-bounds read in the Falcon sensor running in the Windows kernel, and about 8.5 million machines blue-screened. Airlines, hospitals, banks, and 911 centers were hit. Insured losses ran into the billions, and Delta alone claimed about $500M. The RCA and Congressional testimony explained the gap: the sensor’s code was staged and tested, but content updates were validated by a checker with a bug and then deployed globally all at once (crowdstrike.com RCA).
  • Azure Central US, July 30, 2024. A DDoS-defense misconfiguration amplified an attack instead of blocking it, during a summer of repeated Microsoft incidents.
  • OpenAI, December 11, 2024. A new telemetry service overwhelmed the Kubernetes API servers across clusters. DNS caching hid the rollout risk, and engineers were locked out of the control planes they needed to revert. A modern classic: observability tooling as the trigger, published with unusual candor for an AI lab.
  • Google Cloud, June 12, 2025. A new Service Control feature with a null-pointer path, no feature flag, and instant global metadata replication crash-looped API management worldwide (about three hours, ~7.5h for us-central1). Cloudflare (whose Workers KV depended on GCS), Spotify, and dozens of others went down with it. The action items read like this series’ recurring advice: flag-gate everything, stagger global propagation, add backoff.

What the postmortems reveal

“Content” updates are code. CrowdStrike’s split (careful staging for binaries, instant global push for configuration content) is the same pattern as Cloudflare in 2019 and Google in 2025. The hardest-won lesson keeps coming back one level up: anything that changes runtime behavior needs canaries, whether you call it code, config, content, or policy.

Third-party risk moved inside the perimeter. Kernel-resident agents, SaaS platforms, and cloud control planes all showed that your availability is a weighted sum of your vendors’. Vendor rollout policies, kill switches, and “can we run degraded without X?” entered procurement checklists, and, through DORA in the EU, regulation.

Deletion protection became a customer demand. UniSuper (and memories of Atlassian) made soft-delete, multi-party approval for destructive operations, and off-provider backups standard enterprise asks.

AI joined the response, cautiously. By 2025, AI-drafted incident summaries and postmortem first drafts were common in tooling (incident.io, PagerDuty, Rootly), and the debate shifted to whether AI should act, not just summarize.

Practice and tooling shifts

  • Windows resiliency work: Microsoft started pushing security vendors out of the kernel, an OS architecture change driven by a postmortem.
  • Staged rollout for everything became the loudest norm since blameless culture: rings, feature flags, and propagation delays for config and content, not just binaries.
  • Resilience regulation matured. DORA (EU financial sector) took effect in January 2025, making incident reporting and third-party risk management legal obligations.

Takeaways that still hold

  1. Inventory every agent with kernel or root access on your fleet, and know its vendor’s rollout policy before you need to.
  2. Back up your cloud outside that cloud. Assume the account, not just the region, can fail.
  3. If a change can go global in seconds, that speed is itself a risk to engineer around. Add deliberate friction.
  4. Ask every vendor to show you their postmortem culture. It predicts your downtime better than their SLA does.