Patch Notes #106 — Quiet Fortnight, Loud Cluster
No planetary news cycle this window (the universe occasionally rests between chaos sprints), so a working dispatch from the Kubernetes trenches, where the real 2017 is happening. Status: five services migrated, two incidents caused, one genuinely new capability unlocked. The incidents, briefly, because incidents are the tuition: (1) we set memory limits from vibes instead of profiles and the OOMKiller taught us our own traffic patterns; (2) liveness probes pointed at an endpoint that touched the database, so a database blip made Kubernetes “helpfully” restart EVERY healthy pod at once — self-inflicted thundering herd (drink), the automation amplifying the failure it was meant to contain. GitHub’s and Google’s SRE writing warned me about exactly this genre: your resilience machinery is itself a failure domain, and it fails ENTHUSIASTICALLY. Probes now check only what the pod itself controls. ...