Typos That Broke the Internet (Oct 2016 – Dec 2017)

Filed February 4, 2018 — the weekend after GitLab’s live-streamed database recovery turned one.

This is the window that proved honest postmortems build trust instead of costing it. A livestreamed database recovery and a single typo that broke half the web produced two of the most-read incident reports ever written.

The incidents that defined the period

  • Dyn DNS DDoS, October 21, 2016. The Mirai botnet, built from hacked IoT devices, took down a big managed-DNS provider and, with it, Twitter, Netflix, Reddit, and GitHub for most of a day. It was the industry’s introduction to dependency concentration: dozens of “independent” sites all used one DNS provider.
  • GitLab, January 31, 2017. An exhausted engineer fighting replication lag ran rm -rf on the primary’s data directory. Five different backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute writeup (about.gitlab.com). About six hours of data was lost, and GitLab’s reputation arguably went up.
  • AWS S3 us-east-1, February 28, 2017. An operator debugging the billing system mistyped a command and removed far more capacity than intended. The index subsystem needed a full restart it hadn’t done in years (aws.amazon.com/message/41926). Thousands of sites broke, including AWS’s own status page, whose health icons were hosted on S3.
  • Cloudbleed, February 2017. A parser bug leaked memory from one Cloudflare customer into other customers’ cached pages. Cloudflare’s detailed disclosure set a new bar for security postmortems.
  • British Airways, May 2017. A datacenter power event grounded flights worldwide. The vague public explanation became the counterexample to GitLab-style openness.
  • Equifax breach, 2017. An unpatched Struts vulnerability. The lesson was less about the bug and more about knowing what you run and keeping it patched.

What the postmortems reveal

Transparency won, clearly. GitLab and AWS gave specifics: the command, the parameter, the checks they were adding. BA gave vagueness. Customers noticed which companies they trusted more afterward, and “publish the real postmortem” became a strategy rather than a legal risk.

“Human error” stopped counting as a finding. Both of the big incidents were triggered by someone typing, and both writeups put the fault in the system: tooling that allowed a huge capacity removal, backups nobody had test-restored. Sidney Dekker’s line that human error is a symptom, not a cause, became the default view.

Recovery you haven’t tested doesn’t count. GitLab’s five failed backup layers made “we test restores, not backups” a standard action item. S3’s never-restarted index subsystem did the same for cold-start paths.

Practice and tooling shifts

  • Tooling got blast-radius limits. AWS’s own fix, adding floors and rate limits to capacity removal, was the pattern everyone copied.
  • Multi-DNS and multi-CDN setups started showing up in architecture reviews after Dyn.
  • Scheduled, timed restore drills made it into SRE runbooks.

Takeaways that still hold

  1. A backup you haven’t restored is a guess, not a backup.
  2. Any tool that asks “how much capacity to remove” needs a floor, a rate limit, and a confirmation.
  3. Host your status page on someone else’s infrastructure.
  4. Telling the whole story after a failure builds trust. The companies that did it came out ahead.