The SRE Book Era: Error Budgets Meet Cascading Failure

The SRE Book Era (Jul 2015 – Sep 2016) Filed September 20, 2016 — one year to the day after the DynamoDB cascade that defined this window. Google published Site Reliability Engineering in 2016 and gave the industry a shared vocabulary: SLOs, error budgets, toil, blameless postmortems. At the same time, the era’s biggest outages kept showing the same pattern. Systems broke not from the original fault, but from how they tried to recover. ...

September 20, 2016 · July 2015 – September 2016 · Retrospective

Patch Notes #090 — Courage and Combustion

Apple killed the headphone jack. The iPhone 7 has no 3.5mm port, a 138-year-old connector, deprecated on stage, and the marketing chief actually used the word “courage,” which the internet will never let die. The replacement bet is AirPods: $159 wireless earbuds that look like cigarette butts and, per everyone who’s tried them, work like actual magic (the W1 pairing chip is the real product). My take, on the record: in two years the jack outrage will be forgotten and every phone will copy this, because Apple doesn’t remove ports first so much as remove them loudest. (See: floppy, CD drive, every port on the MacBook. The obituary is always premature and always eventually true.) ...

September 11, 2016

Patch Notes #089 — Triple-Triple

Rio closed and the ledger is mythological: Usain Bolt completed the triple-triple, 100m, 200m, 4x100m gold, at three consecutive Olympics. Nine finals, nine golds, zero doubt, all charisma. Phelps retires (again) at twenty-three career golds, more than most countries. Biles owns four golds and the sport’s future. I watched a human run 100 meters in 9.81 seconds while grinning sideways at the field, and I understand now why my non-tech friends tolerate my AlphaGo monologues: excellence in any domain is legible to every domain. ...

August 27, 2016

Patch Notes #088 — Rio and the Sky That Promised Too Much

The Rio Olympics are on. Phelps, at 31, in his fifth Games, is out here winning golds like it’s 2008; Simone Biles is performing gymnastics from a higher difficulty tier than the sport’s scoring anticipated; and the pool has inexplicably turned green (algae plus chemistry mismanagement, and even the Olympics has infra incidents, with status-page communication worse than ours in 2013). The opening ceremony ran on a fraction of London’s budget and was better for the constraint. Constraints breed brilliance; see also: everything. ...

August 12, 2016

Patch Notes #087 — Yahoo, Sold for Parts

Verizon is buying Yahoo’s core business for $4.8 billion. Yahoo. The company that was the internet’s front door when I first touched a browser; the company that turned down buying Google for peanuts and turned down selling to Microsoft for $45B. Sold for the price of a nice office campus, to become part of a telco’s ad play. The kicker every article notes: Yahoo’s stake in Alibaba is worth vastly more than Yahoo itself, so the acquisition they got right outperformed the entire operating company. There’s a lesson about portfolios and a meaner one about focus, and Yahoo is somehow the cautionary tale for both. ...

July 28, 2016

Patch Notes #086 — Gotta Cache 'Em All

Pokémon GO launched one week ago and reality has not recovered. The park across from our office, normally four pigeons and a guy with a guitar, now hosts hundreds of humans at 10pm, phones aloft, hunting a Snorlax. It’s the fastest app to 10M+ downloads anyone can remember (not even officially released in India yet; the APK found us anyway, as APKs do); Nintendo’s market cap grew by tens of billions (for a game they licensed more than made, which the market eventually noticed); and the servers have been down roughly as often as up, because no capacity plan on Earth included “entire human population, simultaneously, outdoors.” ...

July 13, 2016

Patch Notes #085 — The Upset, the Exit, the Brexit

Iceland beat England. Down 1-0 early in the match, the comeback that has never happened in modern English tournament history happened to the most expensive squad of underachievers ever. Final whistle: Iceland’s defensive block materializing out of pure discipline, Sigthorsson’s shot sliding under the goalkeeper, and a nation of 330,000 people celebrating one of the greatest upsets in history. The England manager resigned before the press conference. The group chat’s England fan surfaced this morning staring at the wall and nobody invoked the work queue. The qualification streak is officially the asterisk. Records are vanity metrics. Tournaments are revenue. ...

June 28, 2016

Patch Notes #084 — E3 in the Uncanny Valley

Writing this one with a heavy heart: the Orlando nightclub shooting was yesterday morning and the numbers are unbearable. Tech’s role this time was quiet and human: blood-drive coordination, safety check-ins, families refreshing feeds. Take care of your people. The rest of this entry is the ordinary world continuing, because it does, and that’s both the tragedy and the mercy of it. E3 is on. The consoles blinked first on Moore’s Law: Microsoft announced “Project Scorpio,” a mid-generation more powerful Xbox, and Sony’s Neo is the open secret. The console cycle is becoming a phone cycle, incremental hardware with a shared library. Backward compatibility as a covenant, upgrades as a treadmill. Gaming discovered the annual release cadence right as gaming discovered it can’t finish games by launch day. ...

June 13, 2016

Patch Notes #083 — Game Seven Eve

The Madrid derby clawed back: 1-0 became 1-1, including a Carrasco equalizer that belongs in a museum. The penalty shootout happened last night. The eulogy branch and the comeback branch were both open; one got merged in the 5th round of penalties. Real Madrid won it. The Champions League rematch with Atletico is settled. The universe’s test suite continues. Google I/O delivered as rumored: Google Assistant (their whole company reorganized into a conversation), Google Home (the kitchen-counter war is official), and two chat apps nobody asked for (Allo and Duo, joining Hangouts and Messenger in the graveyard-to-be; Google ships chat apps the way I shipped “fix” commits in 2013). ...

May 29, 2016

Patch Notes #082 — Down 3-1

Sports dispatch: the Champions League final is set, and it’s the Madrid derby again — Real vs Atlético, the 2014 rematch. The group chat has already drafted both the redemption narrative and the heartbreak narrative, committed to separate branches, waiting to see which merges. (Yes, other football happened this month. No, I am not ready to write about the Premier League table yet. Some priors need a few more weeks to die with dignity.) ...

May 14, 2016