When Automation Fights Back (Jan 2018 – Mar 2019)
Filed April 22, 2019 — the first anniversary of the TSB migration weekend.
By 2018, most companies had automated their failover and recovery. The big postmortems this period are about that automation making the wrong call. The question stopped being “why did this break?” and became “why did our own self-healing make it worse?”
The incidents that defined the period
- TSB Bank migration, April 2018. A big-bang core-banking migration locked UK customers out of their accounts for weeks. The independent review that followed became required reading on cutover risk, and regulators started treating operational resilience as something they could enforce.
- GitHub, October 21, 2018. A 43-second network partition between the US East and West Coast datacenters led the orchestration software to promote a West Coast MySQL primary while the East Coast primary still held writes that hadn’t replicated. That’s a split-brain. GitHub chose data consistency over uptime, ran degraded for about 24 hours, and published a strong hour-by-hour writeup (github.blog).
- Microsoft Azure South Central US, September 2018. A lightning strike caused a cooling failure, hardware shut itself down to avoid damage, and the regional outage showed how many “global” Azure services (including Azure AD and the status page) secretly depended on one region.
- Google Cloud, July 2018. A global load-balancing config change briefly broke customers worldwide, reinforcing a growing theme: global control planes mean a global blast radius.
- Facebook, March 13, 2019. A server configuration change caused a roughly 14-hour outage of Facebook, Instagram, and WhatsApp, the longest in the company’s history at the time.
- Wells Fargo, February 2019. A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems.
What the postmortems reveal
Automated failover needs a plan for data. GitHub’s incident became the case study: failover that optimizes for uptime can quietly lose consistency. Postmortems started asking what the orchestrator actually does during a network partition.
Regions aren’t as independent as advertised. Azure’s lightning strike and Google’s load-balancer change both showed control planes, identity, and status pages acting as cross-region coupling points. “Check that this region is really independent” entered DR checklists.
Staying degraded but consistent became a respectable choice. GitHub deliberately took a slower recovery to avoid losing writes, and said so plainly. Postmortems began documenting the decisions and trade-offs made during an incident, not just the causes. The Learning From Incidents community grew up around exactly this.
Practice and tooling shifts
- The SRE Workbook (2018) turned SLO theory into worked examples, and SLO-based alerting started replacing piles of static thresholds.
- Chaos engineering went commercial (Gremlin, ChaosToolkit), and partition testing joined instance-killing.
- Incident command got formal: dedicated incident-commander rotations, severity levels, and comms leads spread from Google and Amazon into mid-size companies.
Takeaways that still hold
- Decide before the partition whether your failover prefers uptime or consistency. Your orchestrator is deciding either way.
- Check which “global” services actually live in one region. Start with auth and your status page.
- Write down the judgment calls made during the incident. The trade-offs teach more than the trigger.