Patch Notes #068 — Autopilot Engaged

Tesla shipped Autopilot this week, an over-the-air update, and suddenly tens of thousands of already-sold cars can steer themselves on highways. Software update as a hardware transformation: the car you bought in June gained a capability in October while parked in your garage. YouTube instantly filled with people testing it in ways the release notes explicitly forbid (hands off, filming, one guy in the back seat), because the gap between “driver assistance” and what marketing lets people hear is a canyon with no lane markings. This will go fine until it doesn’t; noting the date. ...

October 17, 2015

Patch Notes #067 — The Cheat Device Was Software

Volkswagen got caught cheating on emissions tests with code. Eleven million diesel cars shipped with a “defeat device,” software that detected when the car was being tested (steering angle, wheel speed patterns) and switched to clean mode, then polluted up to 40x the legal limit on real roads. The CEO resigned in days; the stock lost a third of its value; criminal probes everywhere. I can’t stop thinking about the engineers. Somewhere, someone wrote if (isEmissionsTest()) and someone else code-reviewed it. Did they push back? Was there a ticket? An email chain? “I was told to” is now a career-ending, possibly indictable, sentence. We got a whole engineering-ethics lunch talk out of it at work, and the line that stuck: your commit history is your signature. Refuse in writing. ...

October 2, 2015

Patch Notes #066 — The Pencil and the Ad-Block Apocalypse

Apple’s fall event: iPhone 6S (“the only thing that’s changed is everything,” sure), a giant iPad Pro with a stylus called Pencil, and the entire internet dug up the 2007 clip of Steve Jobs mocking styluses. Context collapse aside, the Pencil reviews from artists are glowing. Products are allowed to evolve past their founders’ one-liners; companies that can’t do that die with their quotes. The bigger dev story: iOS 9 shipped with support for content blockers, ad blockers, blessed by Apple, on the platform where media companies make their mobile money. Blockers rocketed to #1 on the App Store in hours. Publishers are furious; users are voting with installs; and everyone’s suddenly discussing the web’s original sin: pages where 70% of the bytes are surveillance and ads. Our own site loads 14 trackers. I checked. I’m not proud. ...

September 17, 2015

Patch Notes #065 — Black Monday and Boring Sprints

Markets had a genuinely scary fortnight: China’s slowdown triggered “Black Monday” (Aug 24). The Dow dropped 1,000 points at the open, tech stocks cratered, and for a day the RSU grant I’m about to sign taught me what “paper” means before I’ve vested a single share. The senior folks were fully unbothered: “you can’t sell it anyway, watch the tape less.” By Friday most of it bounced back. Volatility tourism: complete. Lesson: my compensation has a dependency on macroeconomics I cannot patch, monitor, or roll back. Diversification is the only circuit breaker. ...

September 2, 2015

Patch Notes #064 — Google Reboots Itself as Alphabet

Google announced it’s now a subsidiary. Of Alphabet. A new holding company where Google (search, ads, Android) is one division and the moonshots (self-driving cars, life extension, delivery drones, smart cities) get their own letters. Larry and Sergey run Alphabet; Sundar Pichai becomes CEO of Google. The org-chart nerd in me is riveted: this is a refactor. They extracted the profitable monolith from the experimental services, gave each its own P&L, and renamed the parent module. Even the announcement URL was abc.xyz because google.com belongs to the subsidiary now. Conway’s Law says shipping structure mirrors org structure; Alphabet is betting the reverse, that if you restructure the org you change what ships. ...

August 18, 2015

Patch Notes #063 — They Hacked a Jeep on the Highway

Two researchers remotely took over a Jeep, through its internet-connected radio, while a Wired journalist drove it on a highway. Wipers, radio, then the transmission. They killed the engine from a couch ten miles away. Chrysler recalled 1.4 million vehicles, the first mass automotive recall for a software vulnerability. The IoT chickens are coming home and they weigh two tons and drive 70mph. Every device is a computer now, every computer is a target, and the auto industry just learned patch management with the whole world watching. (The fix ships on a USB stick mailed to owners. A USB stick! Mailed!) ...

August 3, 2015

Patch Notes #062 — Pluto, Up Close

New Horizons flew past Pluto. Nine and a half years, three billion miles, and the probe threaded a keyhole in space within 72 seconds of a schedule written when I was in high school. The photos (Pluto has a heart-shaped plain, mountains of ice) arrived at 1-2 kbps, slower than dial-up, and will take over a year to fully download. The spacecraft has less computing power than my phone and less RAM than my Chrome tab about it. Constraints breed brilliance. I printed the heart photo for my desk. ...

July 19, 2015

Patch Notes #061 — Reddit Goes Dark

Reddit revolted this week. The company abruptly fired Victoria, the beloved employee who ran the AMA machinery, and within hours volunteer moderators took hundreds of the biggest subreddits private in protest. The front page of the internet, switched off by its own unpaid workforce. That’s the part that fascinates me: Reddit’s actual product is built and moderated by volunteers, and the company apparently forgot that volunteers can strike. Your community is infrastructure. Treat it like a free CDN and one day the CDN has opinions. ...

July 4, 2015

The SRE Book Era: Error Budgets Meet Cascading Failure

The SRE Book Era (Jul 2015 – Sep 2016) Google published Site Reliability Engineering in 2016 and gave the industry a shared vocabulary: SLOs, error budgets, toil, blameless postmortems. At the same time, the era’s biggest outages kept showing the same pattern. Systems broke not from the original fault, but from how they tried to recover. The incidents that defined the period AWS DynamoDB, September 20, 2015. The classic cascading-failure postmortem (aws.amazon.com/message/5467D2). A network blip made storage servers all re-request their membership data at once. The metadata service was already near capacity from a new feature and couldn’t keep up, and retries made it worse. The outage spread from DynamoDB into EC2, SQS, and CloudWatch in us-east-1. The fixes (more headroom, longer timeouts, smarter retries) read like a distributed-systems checklist. Salesforce NA14, May 2016. A database failure plus a failed failover left a major instance degraded for almost a day, with some data lost for good. It put “your SaaS vendor’s DR plan is your DR plan” into procurement conversations. Southwest (July 2016) and Delta (August 2016). Back-to-back airline meltdowns from single points of failure (a failed router, a datacenter power problem) cancelled thousands of flights. Boards started asking about technical debt. Telstra, 2016. A run of national mobile outages in Australia, one caused by taking a single node offline the wrong way, made the telco postmortem press release routine. What the postmortems reveal Retry storms got a name and a set of standard fixes. The DynamoDB writeup made this failure pattern mainstream well before the academic term “metastable failure” existed. Exponential backoff, jitter, circuit breakers, and load shedding moved out of Netflix blog posts and into default library behavior. ...

July 1, 2015 · July 2015 – September 2016 · Retrospective

Patch Notes #060 — Catalan State of Mind

Barcelona won the treble, their second in history, outlasting a resilient Juventus side in Berlin. Busquets won praise for his defensive positioning, basically, which is like the on-call engineer winning the company award: finally, someone values prevention. The MSN attacking revolution has its trophy now. Every academy and every manager is about to copy it. Success is the most-forked repo. E3 delivered the loudest crowd reaction in gaming history: Final Fantasy VII Remake. Announced. Real. My childhood, re-rendered. Also Fallout 4 went from announcement to “November this year” (a marketing miracle in an industry of eternal pre-orders), and Xbox announced backward compatibility, old games on new consoles, the feature “nobody” asked for except literally everyone. ...

June 19, 2015