Transparency

Typos That Broke the Internet (Oct 2016 – Dec 2017) Filed February 4, 2018 — the weekend after GitLab’s live-streamed database recovery turned one. This is the window that proved honest postmortems build trust instead of costing it. A livestreamed database recovery and a single typo that broke half the web produced two of the most-read incident reports ever written. The incidents that defined the period Dyn DNS DDoS, October 21, 2016. The Mirai botnet, built from hacked IoT devices, took down a big managed-DNS provider and, with it, Twitter, Netflix, Reddit, and GitHub for most of a day. It was the industry’s introduction to dependency concentration: dozens of “independent” sites all used one DNS provider. GitLab, January 31, 2017. An exhausted engineer fighting replication lag ran rm -rf on the primary’s data directory. Five different backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute writeup (about.gitlab.com). About six hours of data was lost, and GitLab’s reputation arguably went up. AWS S3 us-east-1, February 28, 2017. An operator debugging the billing system mistyped a command and removed far more capacity than intended. The index subsystem needed a full restart it hadn’t done in years (aws.amazon.com/message/41926). Thousands of sites broke, including AWS’s own status page, whose health icons were hosted on S3. Cloudbleed, February 2017. A parser bug leaked memory from one Cloudflare customer into other customers’ cached pages. Cloudflare’s detailed disclosure set a new bar for security postmortems. British Airways, May 2017. A datacenter power event grounded flights worldwide. The vague public explanation became the counterexample to GitLab-style openness. Equifax breach, 2017. An unpatched Struts vulnerability. The lesson was less about the bug and more about knowing what you run and keeping it patched. What the postmortems reveal Transparency won, clearly. GitLab and AWS gave specifics: the command, the parameter, the checks they were adding. BA gave vagueness. Customers noticed which companies they trusted more afterward, and “publish the real postmortem” became a strategy rather than a legal risk. ...