Patch Notes #093 — The Day the DNS Died

Friday, someone turned off the internet’s phone book. A massive DDoS hit Dyn, the managed-DNS provider behind Twitter, Netflix, Reddit, Spotify, and GitHub, and for hours, half the household-name web wouldn’t resolve for much of the US. The weapon: Mirai, a botnet of hacked IoT devices. Security cameras. DVRs. Baby monitors. Default passwords, internet-exposed, conscripted by the hundreds of thousands. The IoT prophecy is ahead of schedule: the S in IoT stands for security, and the things we connected without thinking are now infrastructure-grade artillery pointed at whatever choke point volunteers.

The senior-engineer lesson isn’t the botnet, it’s the choke point: dozens of “independent” services shared one DNS provider, so one attack was everyone’s outage. We spent Monday mapping our own hidden singletons: DNS, CDN, email, payments. Redundancy without diversity is bookkeeping. (I have a feeling I’ll be writing that sentence for the rest of my career.)

Also logged this window: Nintendo finally revealed the Switch, a console-handheld hybrid, cartridge slot and all, March launch, and it looks either brilliant or doomed with no middle setting. And the F1 title fight is heading to the final rounds. Rosberg leads by 26 points; Hamilton awaits; the teammate rivalry enters the climax, one champion leaves.

TIL: anycast DNS and why “just add a second provider” involves zone-transfer plumbing nobody rehearsed. We’re rehearsing it now. Friday was the fire drill.