When Automation Fights Back (Jan 2018 – Mar 2019)
By 2018 the industry had automated failover, orchestration, and recovery — and the defining postmortems of this window are about that automation making the wrong call. The question shifted from “why did the component fail?” to “why did our self-healing make it worse?”
The incidents that defined the period
- TSB Bank migration, April 2018 — A big-bang core-banking migration locked UK customers out of accounts for weeks. The subsequent independent review became required reading on cutover risk, and regulators started treating operational resilience as a compliance domain.
- GitHub, October 21, 2018 — 43 seconds of network partition between US East and West Coast datacenters; orchestration software promoted a West Coast MySQL primary while the East Coast primary still held unreplicated writes. Split-brain. GitHub chose data consistency over uptime, running degraded for ~24 hours, and published a superb hour-by-hour analysis (github.blog).
- Microsoft Azure South Central US, September 2018 — A lightning strike caused a cooling failure; hardware shut down to protect itself, and the regional outage revealed how many “global” Azure services (including Azure AD and the status portal) had hidden dependencies on one region.
- Google Cloud, July 2018 — A global load-balancing configuration event briefly broke customers worldwide, feeding a growing theme: global control planes mean global blast radius.
- Facebook, March 13, 2019 — A ~14-hour outage of Facebook, Instagram, and WhatsApp attributed to a server configuration change — at the time the longest outage in the company’s history.
- Wells Fargo, February 2019 — A fire-suppression system triggered a datacenter shutdown, and banking customers lost app and card access. Banks officially had SRE-shaped problems.
What the postmortems reveal
1. Automated failover needs a theory of data. GitHub’s incident became the case study: failover automation that optimizes for availability can silently sacrifice consistency. Postmortems started asking “what does our orchestrator do during a partition?” — a Jepsen-style question applied to ops tooling.
2. Regions aren’t as independent as advertised. Azure’s lightning strike and Google’s global LB event showed control planes, identity, and status pages as cross-region coupling points. “Validate the region-independence claim” entered DR checklists.
3. Degraded-but-consistent became a legitimate choice. GitHub explicitly chose a slower recovery to avoid losing writes, and said so. Postmortems began documenting decisions and trade-offs during the incident, not just causes — a shift toward genuine incident analysis (the LFI / Learning From Incidents community formed around exactly this, drawing on Allspaw’s work).
Practice and tooling shifts
- The SRE Workbook (2018) landed, turning SLO theory into worked examples; SLO-based alerting started replacing threshold soup.
- Chaos engineering commercialized (Gremlin, ChaosToolkit) and “partition testing” joined instance-killing.
- Incident command formalized: dedicated IC rotations, severity matrices, and comms leads spread from Google/Amazon practice into mid-size companies.
Takeaways that still hold
- Decide before the partition whether your failover prefers availability or consistency — your orchestrator is deciding it either way.
- Audit which “global” services actually live in one region (start with auth and your status page).
- Write down the judgment calls made during the incident; the trade-offs are more instructive than the trigger.