Shared Fate in the Cloud (Apr 2014 – Jun 2015) Filed July 12, 2015 — the weekend after the NYSE, United Airlines, and the WSJ all fell over on the same Wednesday. This is when everyone learned that moving to the cloud means sharing your provider’s mistakes too: their patches, their config pushes, their operators' typos. Security incidents also started getting the same careful postmortem treatment as plain downtime. The incidents that defined the period Heartbleed, April 2014. The OpenSSL bug that forced mass certificate rotation across the internet. The real operational lesson was that almost nobody had a list of where TLS actually terminated, so “patch and rotate” took weeks. Shellshock in September repeated the drill for bash. Joyent, May 2014. An operator running a routine update rebooted an entire data center of customer systems with one command. Joyent’s postmortem was direct: the problem wasn’t the operator, it was that the tooling let you target a whole datacenter with no confirmation. AWS Xen reboot, September 2014. AWS rebooted a large share of EC2 instances to patch a Xen bug before it was disclosed. Customers who had designed for instance failure barely noticed. Those who hadn’t found out the hard way which servers they’d been treating as pets. Microsoft Azure Storage, November 2014. A performance fix went out globally, skipping the usual staged rollout, and an infinite loop in the blob frontends took down storage across regions. Microsoft’s postmortem admitted the team had gone around its own rollout policy. It’s one of the most-cited config-change postmortems there is. NYSE, United Airlines, and the WSJ, July 8, 2015. Three unrelated same-day outages that everyone assumed were connected. A good example of how reliability failures turn into a news cycle. What the postmortems reveal Configuration changes became the leading cause. The Azure writeup captured a pattern that still dominates postmortems: the code was fine, and the rollout of a config flag was what broke things. “All deploys are staged, including config” started showing up in action items. ...

Reddit revolted this week. The company abruptly fired Victoria, the beloved employee who ran the AMA machinery, and within hours volunteer moderators took hundreds of the biggest subreddits private in protest. The front page of the internet, switched off by its own unpaid workforce. That’s the part that fascinates me: Reddit’s actual product is built and moderated by volunteers, and the company apparently forgot that volunteers can strike. Your community is infrastructure. Treat it like a free CDN and one day the CDN has opinions. ...

Barcelona won the treble, their second in history, outlasting a resilient Juventus side in Berlin. Busquets won praise for his defensive positioning, basically, which is like the on-call engineer winning the company award: finally, someone values prevention. The MSN attacking revolution has its trophy now. Every academy and every manager is about to copy it. Success is the most-forked repo. E3 delivered the loudest crowd reaction in gaming history: Final Fantasy VII Remake. Announced. Real. My childhood, re-rendered. Also Fallout 4 went from announcement to “November this year” (a marketing miracle in an industry of eternal pre-orders), and Xbox announced backward compatibility, old games on new consoles, the feature “nobody” asked for except literally everyone. ...

The sports story of the fortnight happened in a hotel at dawn: Swiss police raided FIFA, arresting officials on US corruption charges, decades of bribes around World Cup hosting and marketing rights. Then Sepp Blatter won re-election anyway, then resigned four days later as the walls closed in. The organization running the sport I learned to love during Brazil 2014 was apparently corrupt at the protocol layer. The engineers’ lunch take: FIFA had no code review, no audits, and root access for thirty years. Governance is infrastructure. It fails like infrastructure. ...

Quieter fortnight, so a rare inward-looking one. I’m closing in on four years in the industry this summer, and in this week’s review my manager used the word “senior,” as in “the trajectory toward,” and I walked out vibrating slightly. Then I reread this blog from #001 and did my own review. The kid who needed git reflog explained now explains it. The kid terrified of code review now runs them. But also: I still Google the flag order for tar every single time, still get imposter syndrome on Sundays, still write “fix” commits at 6pm. Maybe seniority isn’t the absence of that stuff. Maybe it’s just a longer list of things that don’t scare you anymore. ...

Mayweather-Pacquiao finally happened Saturday, five years of hype, $100 pay-per-view, the biggest fight of the century, and the story of the night was infrastructure. The PPV systems buckled under record demand; the fight got delayed about 45 minutes so buyers could get through; Periscope and Meerkat filled with thousands of people just streaming their TVs, while the providers’ checkout pages 500’d. The boxing was cautious and forgettable. The load test was historic. Two brand-new livestreaming apps just showed an entire industry its own future and its own fragility in one night. ...

Apple Watch preorders shipped this week-ish (Apr 24), and one of our designers got one day-one (imported via a cousin in the Valley — the traditional Indian supply chain). Office consensus after passing it around: gorgeous, laggy, and nobody, including Apple, whose ads show it doing forty things, knows what it’s for yet. My favorite genre of product: shipped confident, purpose TBD. The iPad was this too and figured itself out. The watch gets two years of patience from me. ...

Amazon announced the Dash Button, a physical WiFi button you stick on your washing machine that orders more detergent when pressed, on March 31st, and the entire internet assumed it was an early April Fools joke. It’s real. That’s the tell for this era: the real products and the parody products have converged. A button that buys one thing is either the dumbest object ever shipped or a genuinely clever removal of every friction between want and buy, and knowing Amazon, they’ve A/B tested which. ...

Apple’s Watch event happened, prices from $349 to literally $10,000 for the gold one. Apple selling a five-figure fashion object with the same chip as the cheap one is the boldest margin experiment in consumer tech history. The dev question that matters: what’s an app that’s good on a wrist? Glances, not sessions. Notifications that deserve you. Nobody’s cracked it yet, including Apple. SXSW belonged to an app called Meerkat: live-stream from your phone, one tap. Then Twitter, who owns a competitor called Periscope they haven’t even launched yet, cut Meerkat’s access to the social graph mid-conference. Platform risk in real time, at the same festival where everyone learned Meerkat’s name. Google Reader taught me this lesson as a user; Meerkat is learning it as a company: build on someone’s platform, live at their pleasure. ...

Two things happened on the same day last week (Feb 26) and I need you to appreciate the range: the FCC passed strong net neutrality (Title II, the real thing, the four-million-comments thing, the biggest internet policy win of my lifetime), and the entire planet simultaneously stopped working over a photo of a dress. White and gold or blue and black. Our office split 60/40 and got loud about it. (It’s blue and black. The lighting is doing the lying. Color constancy is the brain’s cache invalidation bug.) ...