There’s an old joke in operations: the monitoring system is the most dangerous thing in the data centre. On 11 December OpenAI proved it, very publicly, and then published one of the clearer post-incident write-ups I’ve read this year.
What Happened
Between 3:16pm and 7:38pm PST, ChatGPT, the API and Sora were degraded or completely unavailable. The trigger was a new telemetry service meant to improve observability of OpenAI’s Kubernetes clusters:
- It ran fine in staging on 10 December.
- In production, on much larger clusters, its configuration caused resource-intensive Kubernetes API operations from every node at once. The API servers were overwhelmed.
- Many services depend on the Kubernetes control plane for DNS-based service discovery. When it fell over, services couldn’t find each other.
- DNS caching hid the problem at first, so the rollout carried on to more clusters before anyone could see the damage.
- Fixing it was hard because the overloaded control plane also locked engineers out of the very tools they needed to roll back.
Lessons Worth Stealing
- Your staging environment is lying to you about scale. If a change’s impact grows with cluster size, test at production size, or roll out cluster by cluster with automatic halts.
- Caches delay feedback. They make systems resilient, but they also hide failures until it’s too late. Build health signals that don’t depend on cached data.
- Plan for being locked out. Break-glass access to the control plane should work when the control plane is the thing that’s broken.
- Know your hidden dependencies. “The data plane keeps running without the control plane” is only true until DNS gets involved.
As an interesting aside, the change was designed to improve reliability. Plenty of outages start that way. Changes to observability, security and safety tooling deserve the same caution as anything else, because they often run everywhere, with high privileges.
