<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Crowdstrike on Azarudeen.com</title><link>http://azarudeen.com/tags/crowdstrike/</link><description>Recent content in Crowdstrike on Azarudeen.com</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Mon, 01 Apr 2024 10:00:00 +0530</lastBuildDate><atom:link href="http://azarudeen.com/tags/crowdstrike/index.xml" rel="self" type="application/rss+xml"/><item><title>The CrowdStrike Reckoning: Third-Party Risk Becomes Everyone's Root Cause</title><link>http://azarudeen.com/posts/10-apr-2024-to-jun-2025-the-crowdstrike-reckoning/</link><pubDate>Mon, 01 Apr 2024 10:00:00 +0530</pubDate><guid>http://azarudeen.com/posts/10-apr-2024-to-jun-2025-the-crowdstrike-reckoning/</guid><description>&lt;h1 id="the-crowdstrike-reckoning-apr-2024--jun-2025"&gt;The CrowdStrike Reckoning (Apr 2024 – Jun 2025)&lt;/h1&gt;
&lt;p&gt;One Friday in July 2024 produced the largest IT outage in history — and it
wasn&amp;rsquo;t a cloud provider. This window&amp;rsquo;s postmortems are dominated by &lt;em&gt;other
people&amp;rsquo;s software&lt;/em&gt; running inside your trust boundary: security agents in the
kernel, a dealer platform for an entire industry, a cloud vendor deleting a
customer, and a quota policy pushed worldwide.&lt;/p&gt;
&lt;h2 id="the-incidents-that-defined-the-period"&gt;The incidents that defined the period&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Google Cloud / UniSuper, May 2024&lt;/strong&gt; — A misconfiguration during provisioning
led Google Cloud to &lt;strong&gt;delete an entire customer&amp;rsquo;s private cloud subscription&lt;/strong&gt;
— a ~$125B pension fund — causing ~two weeks of disruption. Recovery leaned on
UniSuper&amp;rsquo;s own third-party backups. The joint apology statement was
unprecedented; &amp;ldquo;what if our cloud account itself is the failure domain?&amp;rdquo;
entered every DR review.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;CDK Global, June 2024&lt;/strong&gt; — Ransomware took down the SaaS platform underpinning
~15,000 North American car dealerships for weeks. A whole industry discovered
it had a single point of failure it had never load-tested: its vendor.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;CrowdStrike, July 19, 2024&lt;/strong&gt; — A faulty &lt;strong&gt;Rapid Response Content&lt;/strong&gt; update
(Channel File 291) hit an out-of-bounds read in the Falcon sensor running in
the Windows kernel: ~8.5 million machines blue-screened. Airlines, hospitals,
banks, 911 centers. Insured losses in the billions; Delta alone claimed ~$500M.
The RCA and Congressional testimony detailed the gap: sensor &lt;em&gt;code&lt;/em&gt; was staged
and tested; &lt;em&gt;content&lt;/em&gt; updates were validated by a checker with a bug and
deployed globally at once
(&lt;a href="https://www.crowdstrike.com/falcon-content-update-remedial-and-preventative-actions/"&gt;crowdstrike.com RCA&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Azure Central US, July 30, 2024&lt;/strong&gt; — A DDoS defense misconfiguration amplified
rather than mitigated an attack, in a summer of repeated Microsoft incidents.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;OpenAI, December 11, 2024&lt;/strong&gt; — A new telemetry service overwhelmed Kubernetes
API servers across clusters; DNS caching masked the rollout risk, and engineers
were &lt;strong&gt;locked out of the control planes they needed to revert&lt;/strong&gt;. A modern
classic: observability tooling as the outage trigger, published with unusual
candor for an AI lab.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Google Cloud, June 12, 2025&lt;/strong&gt; — A new Service Control policy feature with a
&lt;strong&gt;null-pointer path, no feature flag, and instant global metadata replication&lt;/strong&gt;
crash-looped API management worldwide (~3 hours; ~7.5h for us-central1).
Cloudflare (whose Workers KV depended on GCS), Spotify, and dozens of others
went down with it. The postmortem&amp;rsquo;s own action items read like this series'
greatest hits: flag-gate everything, stagger global propagation, add backoff.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="what-the-postmortems-reveal"&gt;What the postmortems reveal&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;1. &amp;ldquo;Content&amp;rdquo; updates are code.&lt;/strong&gt; CrowdStrike&amp;rsquo;s split — rigorous staging for
binaries, instant global push for configuration content — is the same pattern as
Cloudflare 2019 and Google 2025. The industry&amp;rsquo;s hardest-won lesson keeps
recurring one abstraction level up: &lt;em&gt;anything that changes runtime behavior
needs canaries&lt;/em&gt;, whether it&amp;rsquo;s called code, config, content, or policy.&lt;/p&gt;</description></item></channel></rss>