Typos That Broke the Internet: S3, GitLab, and Radical Transparency

Sat, 01 Oct 2016 10:00:00 +0530

Typos That Broke the Internet (Oct 2016 – Dec 2017)

If one window proved that public, honest postmortems build more trust than they cost, it’s this one. A livestreamed database recovery and a typo that took down half the web produced two of the most-read incident reports in history.

The incidents that defined the period

Dyn DNS DDoS, October 21, 2016 — The Mirai botnet, built from IoT devices, took down a major managed-DNS provider and with it Twitter, Netflix, Reddit, and GitHub for much of a day. The industry’s introduction to dependency concentration: dozens of “independent” sites shared one DNS provider.
GitLab database incident, January 31, 2017 — An exhausted engineer, fighting replication lag, ran rm -rf on the primary’s data directory. Five backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute postmortem (about.gitlab.com). ~6 hours of data was lost — and GitLab’s reputation arguably improved.
AWS S3 us-east-1, February 28, 2017 — An operator debugging the billing system mistyped a playbook parameter and removed far more capacity than intended; the index subsystem required a full restart it hadn’t had in years (aws.amazon.com/message/41926). Thousands of sites broke — including, memorably, AWS’s own status page, whose health icons were hosted on S3.
Cloudbleed, February 2017 — A parser bug leaked memory across Cloudflare customers into cached pages. Cloudflare’s forensic-grade disclosure set a new bar for security postmortems.
British Airways, May 2017 — A datacenter power event (a contractor and a UPS) grounded flights globally; the vague public explanation became the counterexample to GitLab-style transparency.
Equifax breach, 2017 — An unpatched Struts vulnerability; the postmortem lesson was less about the bug than about asset inventory and patch governance.

What the postmortems reveal

1. Transparency won, decisively. GitLab and AWS gave specifics (the command, the parameter, the safety checks now added); BA gave vagueness. The market noticed which companies it trusted more afterward. “Publish the real postmortem” became a competitive strategy, not a legal risk.

Gitlab on Azarudeen.com

Typos That Broke the Internet: S3, GitLab, and Radical Transparency

Typos That Broke the Internet (Oct 2016 – Dec 2017)

The incidents that defined the period

What the postmortems reveal