Patch Notes #106 — Quiet Fortnight, Loud Cluster

No planetary news cycle this window (the universe occasionally rests between chaos sprints), so a working dispatch from the Kubernetes trenches, where the real 2017 is happening. Status: five services migrated, two incidents caused, one genuinely new capability unlocked. The incidents, briefly, because incidents are the tuition: (1) we set memory limits from vibes instead of profiles and the OOMKiller taught us our own traffic patterns; (2) liveness probes pointed at an endpoint that touched the database, so a database blip made Kubernetes “helpfully” restart EVERY healthy pod at once — self-inflicted thundering herd (drink), the automation amplifying the failure it was meant to contain. GitHub’s and Google’s SRE writing warned me about exactly this genre: your resilience machinery is itself a failure domain, and it fails ENTHUSIASTICALLY. Probes now check only what the pod itself controls. ...

May 9, 2017

Patch Notes #105 — The $400 Bag Squeezer

Bloomberg performed the tech journalism of the decade: they squeezed a Juicero bag BY HAND. Backstory for the future readers of this archive: Juicero raised ~$120M from top-tier VCs for a $400 (originally $700!) WiFi-connected juice press that squeezes proprietary $7 produce bags — DRM’d juice, QR-verified, subscription-locked. The machine is, by teardown consensus, a gorgeous over-engineered marvel of custom machining. And Bloomberg demonstrated that human hands squeeze the bags just as well. Faster, actually. ...

April 24, 2017

Patch Notes #104 — Re-Accommodated

Today United Airlines had passengers watch security drag a seated, paying customer off an overbooked flight — bloodied, filmed from four angles, viral before the plane took off. The CEO’s first response apologized for having to “re-accommodate” customers, a word that instantly entered the corporate-euphemism hall of fame next to “sunsetting” and “rightsizing.” By tomorrow the stock will pay for it and the apology will iterate. Filed for the craft because it’s an INCIDENT RESPONSE case study: the event was bad; the response made it catastrophic. First statement matters most; euphemism reads as contempt; and any policy that ends with “and then we call security on our own customer” was a bug long before it executed. Every company has policies like this — dormant, absurd, waiting for their viral moment. Audit for them the way we audit for Shellshock. ...

April 9, 2017

Patch Notes #103 — Demo Day at Scale

YC Demo Day this week: ~100 companies, and the batch reads like a map of 2017’s ambitions — AI-for-everything (radiology, contracts, crops), a heavy fintech contingent aiming at everything banks do badly, and a few “hard tech” moonshots. Four years ago (#006) the joke was “Airbnb for boats.” My marina-logistics friend from that era? His “boring, profitable” pivot (#040) just raised a Series A. The joke companies died; the spreadsheet-shaped ones compounded. Demo Day is a portfolio-priors update on schedule: whatever’s overrepresented in March is oversaturated by August, and the winner is usually filed under “wait, that’s a company?” ...

March 25, 2017

Patch Notes #102 — The Typo and the Triumph

The great S3 outage of 2017: on February 28th, an AWS engineer debugging the billing system typo’d a playbook command and removed WAY more capacity than intended from us-east-1’s S3 — and it turned out half the internet, including AWS’s OWN STATUS DASHBOARD, lives in that bucket-shaped basket. Four hours of a broken web. Amazon’s postmortem is admirably specific: the tool allowed too-big removals (now capped), and a subsystem hadn’t been restarted in YEARS and took ages to cold-boot. Zero blame placed on the human, all of it on the system that let one keystroke go nuclear — the Joyent lesson (2014, #032-adjacent) at 100x scale. If a typo can take down the internet, the typo isn’t the bug. Also: HOST YOUR STATUS PAGE SOMEWHERE ELSE. Everyone. Please. ...

March 10, 2017

Patch Notes #101 — The Blog Post That Shook an Industry

The most consequential engineering document of the fortnight wasn’t a paper or a postmortem — it was a personal blog post. Susan Fowler, an engineer, published “Reflecting on One Very, Very Strange Year at Uber”: calm, precise, receipts-grade documentation of harassment reported and buried, HR processes that protected the wrong people, and a culture rotting from its incentives. It detonated. Uber launched an external investigation within days; the industry’s whisper networks went loud; and every eng leadership team I know — including ours, forty people, no excuses — is asking “could that happen here and would we know?” The technical-adjacent lesson: she wrote it like an incident report — timeline, evidence, escalation paths, systemic analysis over villain-hunting — and THAT is why it couldn’t be dismissed. Documentation is power. It always was. ...

February 23, 2017

Patch Notes #100 — 28-3

ENTRY ONE HUNDRED. Four years and a month of fortnightly patch notes, and the office spent the milestone week debating the greatest comebacks in sports history: we kept returning to Manchester United’s 1999 Champions League final. Down 1-0 in the 90th minute — win probability models would have shown Bayern Munich above 99% — and then the most dramatic three minutes in football history, sealed before the whistle. Teddy Sheringham sweeping in the equalizer, Ole Gunnar Solskjær stabbing the winner into the roof of the net like a man defying the concept of physics out of spite. ...

February 8, 2017

Patch Notes #099 — $299, March 3, Breath of the Wild

Switch details landed: $299, March 3rd, and the new Zelda — Breath of the Wild — ships DAY ONE. Preorder: executed within ninety seconds of the stream ending. The lineup beyond Zelda is thin and the paid-online announcement stung, but Nintendo is doing the thing great engineering orgs do after a failure: the Wii U postmortem is visible in every Switch decision. Confusing message? Now one clear concept (the console you take with you). Starved third-party support? Cartridges, portable dev kits, unreal engine support. Weak launch title? THE most anticipated Zelda ever. You can read the incident review in the product. That’s how you know it was a real review and not theater. ...

January 24, 2017

Patch Notes #098 — Alexa Ate CES

Year five begins. CES verdict from every floor report: ALEXA WON, and Amazon barely even had a booth. Alexa in fridges, cars, lamps, showerheads — hundreds of third-party devices, all voice-piping to Amazon’s cloud. While Google and Apple were building assistants as FEATURES of their own hardware, Amazon quietly built one as a PLATFORM and let everyone else do the manufacturing. The Echo looked like a gimmick in 2014; it’s an ecosystem in 2017. The platform play is invisible until it’s inevitable (Proverbs file, entry 121 — I’ve started numbering them in the blog since apparently this is a permanent institution). ...

January 9, 2017

Patch Notes #097 — Year Four Retrospective: The Year of Broken Priors

Ninety-seven entries. Four years, zero missed. The streak is my longest-running production system. 2016’s THESIS, and it wasn’t subtle: every model broke. Leicester City won the Premier League at 5000-to-1 — the single least probable result in team sports history, and I never even blogged it because I didn’t BELIEVE it in May. Hamilton lost the title despite a rain-delayed heroic run (#094) as Rosberg took the crown and retired five days later. England fell to Iceland’s defensive block (#085). Brexit beat the polls; the election beat the polls; AlphaGo beat the species. If your priors survived 2016 intact, you weren’t paying attention or you weren’t betting. The out-of-distribution events are the only ones that matter (#085), and this year was ALL distribution tails, all the time. ...

December 25, 2016