Real Madrid won the Champions League quarter-final first leg against Juventus on Tuesday, their victory sealed via an unconscionable bicycle kick from Cristiano Ronaldo that had the Turin crowd standing to applaud. My prediction methodology this year (seeding by club’s median engineering team size) survived to the quarter-finals, a personal record for the science. The tournament’s lesson never changes and I never tire of it: cup football is variance worship, and variance is why we watch. ...

The heavy fortnight arrived on schedule (the archive’s rhythm never misses). Cambridge Analytica: the Observer and NYT detailed how a researcher’s personality-quiz app harvested data on ~50 million Facebook users, the quiz-takers and their entire friend graphs, via a permissions model that was working as designed in 2014, and how that corpus flowed to a political-targeting firm. The feed reckoning (“those are our systems”) has its named scandal now. The API breadth was the bug; “we changed it in 2015” is true and insufficient; institutional scrutiny has found the feed. Zuckerberg’s been summoned by three governments this week. The stock shed ~$50B. Every “move fast” platform decision from 2010-2014 is getting re-litigated with 2018 stakes, and honestly, it should be. Data you share is data you can’t unshare; the friend-graph made consent transitive without asking the friends. Architecture is policy. ...

Olympics closed (Norway won the medal table with a population smaller than the Bay Area, depth-over-stars as national sports architecture; their stated youth-sports policy bans scoring before age 13, and I’ve added it to the engineering-culture file under “delay the metrics, develop the fundamentals”). A quiet news fortnight otherwise, the kind the archive shows always precedes a loud one, so let me use it on the two slow-burn projects defining my quarter. ...

Grading last entry’s predictions, as promised. Went 0-for-2 in the best possible way. Spurs drew in Turin. Not Juve-by-2, but a 2-2 shootout where Tottenham came back from 2-0 down, capped by a tactical trick: a backup midfielder dropping deep to feed Kane on a decoy run the manager installed for exactly this moment. Champions League-shaking audacity from the understudy. The away end in Turin was jubilant, gloriously. Backup systems, properly drilled, can outperform the primary (the MTTR sermon in cleats). ...

Typos That Broke the Internet (Oct 2016 – Dec 2017) Filed February 4, 2018 — the weekend after GitLab’s live-streamed database recovery turned one. This is the window that proved honest postmortems build trust instead of costing it. A livestreamed database recovery and a single typo that broke half the web produced two of the most-read incident reports ever written. The incidents that defined the period Dyn DNS DDoS, October 21, 2016. The Mirai botnet, built from hacked IoT devices, took down a big managed-DNS provider and, with it, Twitter, Netflix, Reddit, and GitHub for most of a day. It was the industry’s introduction to dependency concentration: dozens of “independent” sites all used one DNS provider. GitLab, January 31, 2017. An exhausted engineer fighting replication lag ran rm -rf on the primary’s data directory. Five different backup mechanisms failed or were misconfigured. GitLab livestreamed the recovery on YouTube and published a minute-by-minute writeup (about.gitlab.com). About six hours of data was lost, and GitLab’s reputation arguably went up. AWS S3 us-east-1, February 28, 2017. An operator debugging the billing system mistyped a command and removed far more capacity than intended. The index subsystem needed a full restart it hadn’t done in years (aws.amazon.com/message/41926). Thousands of sites broke, including AWS’s own status page, whose health icons were hosted on S3. Cloudbleed, February 2017. A parser bug leaked memory from one Cloudflare customer into other customers’ cached pages. Cloudflare’s detailed disclosure set a new bar for security postmortems. British Airways, May 2017. A datacenter power event grounded flights worldwide. The vague public explanation became the counterexample to GitLab-style openness. Equifax breach, 2017. An unpatched Struts vulnerability. The lesson was less about the bug and more about knowing what you run and keeping it patched. What the postmortems reveal Transparency won, clearly. GitLab and AWS gave specifics: the command, the parameter, the checks they were adding. BA gave vagueness. Customers noticed which companies they trusted more afterward, and “publish the real postmortem” became a strategy rather than a legal risk. ...

A rare cliffhanger entry: this fortnight is all eves, and I’m logging predictions before outcomes so the record can grade me. Next week: Champions League Round of 16, Tottenham vs Juventus. Spurs are starting their backup defender against the Italian defensive dynasty; the fanbase has braced itself in anticipatory self-defense. Prediction: Juve by 2, with maximum pain. (I want Spurs to advance. Wanting is not forecasting.) Tuesday: SpaceX attempts the first Falcon Heavy launch, twenty-seven engines, three boosters, the most powerful rocket since the Saturn V, with Elon’s personal Tesla Roadster as the dummy payload because the payload-mass simulator rules don’t say it can’t be a car. Musk himself puts odds at coin-flip and says success is “not blowing up the pad.” The two side boosters are supposed to land back, together, synchronized. Prediction: it flies, one booster sticks. (The kid who cried at one landing is emotionally unprepared for two.) ...

Last Saturday morning, every phone in Hawaii received: “BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.” For thirty-eight minutes, it stood. People put children in storm drains. Said goodbye. And the cause, when it emerged, was the most professionally humbling artifact of the year: an employee running a shift-change drill selected the wrong item from a dropdown menu. The real-alert option lived adjacent to the test option, same list, similar labels, with a confirmation dialog that (like all confirmation dialogs) had been trained into muscle-memory clickthrough. No cancel template existed; one had to be composed, mid-crisis, over 38 minutes. ...

Year six opens with the most unsettling vulnerability disclosure I’ve ever read: Meltdown and Spectre. Not bugs in software, but bugs in the idea of modern CPUs. Speculative execution, the trick where processors guess ahead to go fast, turns out to leak secrets through timing side channels. Meltdown lets a process read kernel memory; Spectre tricks other processes into leaking their own; between them, essentially every Intel chip since the 90s and most others are affected. The fix costs performance (the cloud providers are rebooting the entire planet’s fleet this week, imagine that change-management ticket), and Spectre-class attacks will haunt chip design for a decade because the flaw is load-bearing: the speed we’ve enjoyed since 1995 was partially borrowed against an invariant nobody wrote down. ...

Entry 121, year five complete, streak intact. This one’s dated the 20th because the 25th belongs to family and the 30th to Zelda; seniority is knowing your own load limits. The closing fortnight compressed the whole year: net neutrality was repealed on the 14th as scheduled (the slow-boil forecast now on the clock); Bitcoin kissed ~$19,783 on the 17th and is already wobbling (if that was the top, let the record show the top smelled like my barber’s price targets); and The Last Jedi opened to a fascinating split, critics elated, a vocal fan-segment furious, which mostly taught me that beloved legacy systems can’t be refactored without someone filing a grievance about the original architecture. (I liked it. The Luke arc is the brave design choice.) ...

The most 2017 sentence ever written, and I get to write it: trading of cartoon cats has congested the Ethereum network. CryptoKitties (collectible, breedable, blockchain-native cats) launched last week and immediately became the largest consumer of gas on the “world computer,” at points comprising 15-20%+ of all network traffic, backing up transactions globally and spiking fees for everyone trying to do, you know, finance. People have paid over $100,000 for a single virtual cat. The serious take hiding in the absurdity: this is the first organic consumer dapp product-market fit, and it instantly hit the scaling wall. Ethereum does ~15 transactions per second, total, planet-wide, and one viral toy saturated it (the whole chain is one hot partition, by design). Every scaling roadmap conversation (sharding, layer-2, state channels) just got its forcing function, and its mascot. Thundering herds (drink) now come in kitten form. ...