When Automation Fights Back: Split Brains, Lightning Strikes, and SLOs at Scale
When Automation Fights Back (Jan 2018 – Mar 2019) Filed April 22, 2019 — the first anniversary of the TSB migration weekend. By 2018, most companies had automated their failover and recovery. The big postmortems this period are about that automation making the wrong call. The question stopped being “why did this break?” and became “why did our own self-healing make it worse?” The incidents that defined the period TSB Bank migration, April 2018. A big-bang core-banking migration locked UK customers out of their accounts for weeks. The independent review that followed became required reading on cutover risk, and regulators started treating operational resilience as something they could enforce. GitHub, October 21, 2018. A 43-second network partition between the US East and West Coast datacenters led the orchestration software to promote a West Coast MySQL primary while the East Coast primary still held writes that hadn’t replicated. That’s a split-brain. GitHub chose data consistency over uptime, ran degraded for about 24 hours, and published a strong hour-by-hour writeup (github.blog). Microsoft Azure South Central US, September 2018. A lightning strike caused a cooling failure, hardware shut itself down to avoid damage, and the regional outage showed how many “global” Azure services (including Azure AD and the status page) secretly depended on one region. Google Cloud, July 2018. A global load-balancing config change briefly broke customers worldwide, reinforcing a growing theme: global control planes mean a global blast radius. Facebook, March 13, 2019. A server configuration change caused a roughly 14-hour outage of Facebook, Instagram, and WhatsApp, the longest in the company’s history at the time. Wells Fargo, February 2019. A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems. What the postmortems reveal Automated failover needs a plan for data. GitHub’s incident became the case study: failover that optimizes for uptime can quietly lose consistency. Postmortems started asking what the orchestrator actually does during a network partition. ...