Typos That Broke the Internet (Oct 2016 – Dec 2017)

If one window proved that public, honest postmortems build more trust than they cost, it’s this one. A livestreamed database recovery and a typo that took down half the web produced two of the most-read incident reports in history.

The incidents that defined the period

  • Dyn DNS DDoS, October 21, 2016 — The Mirai botnet, built from IoT devices, took down a major managed-DNS provider and with it Twitter, Netflix, Reddit, and GitHub for much of a day. The industry’s introduction to dependency concentration: dozens of “independent” sites shared one DNS provider.
  • GitLab database incident, January 31, 2017 — An exhausted engineer, fighting replication lag, ran rm -rf on the primary’s data directory. Five backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute postmortem (about.gitlab.com). ~6 hours of data was lost — and GitLab’s reputation arguably improved.
  • AWS S3 us-east-1, February 28, 2017 — An operator debugging the billing system mistyped a playbook parameter and removed far more capacity than intended; the index subsystem required a full restart it hadn’t had in years (aws.amazon.com/message/41926). Thousands of sites broke — including, memorably, AWS’s own status page, whose health icons were hosted on S3.
  • Cloudbleed, February 2017 — A parser bug leaked memory across Cloudflare customers into cached pages. Cloudflare’s forensic-grade disclosure set a new bar for security postmortems.
  • British Airways, May 2017 — A datacenter power event (a contractor and a UPS) grounded flights globally; the vague public explanation became the counterexample to GitLab-style transparency.
  • Equifax breach, 2017 — An unpatched Struts vulnerability; the postmortem lesson was less about the bug than about asset inventory and patch governance.

What the postmortems reveal

1. Transparency won, decisively. GitLab and AWS gave specifics (the command, the parameter, the safety checks now added); BA gave vagueness. The market noticed which companies it trusted more afterward. “Publish the real postmortem” became a competitive strategy, not a legal risk.

2. “Human error” was declared a non-finding. Both marquee incidents were triggered by a person typing, and both writeups located the fault in the system: tools that allowed too-large capacity removals, backups that were never restore- tested. Sidney Dekker’s “human error is a symptom, not a cause” became postmortem orthodoxy.

3. Untested recovery = no recovery. GitLab’s five failed backup layers made “we test restores, not backups” a standard action item. S3’s never-restarted index subsystem did the same for cold-start paths.

Practice and tooling shifts

  • Blast-radius limits in tooling: AWS’s own action item — capacity removal now has floors and rate limits — became the pattern everyone copied.
  • Multi-DNS and multi-CDN strategies appeared in architecture reviews post-Dyn.
  • Game-day restores: scheduled, timed database restore drills entered SRE runbooks.

Takeaways that still hold

  1. A backup you haven’t restored is a hypothesis, not a backup.
  2. Any tool that accepts “how much capacity to remove” needs a floor, a rate limit, and a confirmation.
  3. Host your status page on someone else’s infrastructure.
  4. Radical transparency after failure is a trust-building move — the companies that told the whole story came out ahead.