Four weeks after AWS’s DNS race condition, it was Cloudflare’s turn. From 11:20 UTC on Tuesday 18 November, sites behind Cloudflare, including X and ChatGPT, started returning errors around the world. Cloudflare’s CEO published a detailed write-up the same day, and described it as the company’s worst outage since 2019.
What Happened
- Cloudflare’s Bot Management system scores every request to decide whether it comes from a human or a bot. Think of it as the bouncer on the door. It relies on a “feature file” that’s regenerated every few minutes and pushed across the network.
- A permissions change on a ClickHouse database changed the results of the query that builds that file. It started returning duplicate rows.
- The feature file doubled in size and went over a hard limit of 200 features in the proxy software. The proxy treated that as fatal, and requests started failing with 5xx errors.
- Because the file was rebuilt every five minutes, and only some database nodes had the change, the network flickered between good and bad. Early on, that looked like an attack, which sent investigators in the wrong direction for a while.
- At 14:24 UTC Cloudflare stopped generating new files and pushed a known-good version. Core traffic recovered soon after, with full recovery later that afternoon.
The Pattern Is Now Unmistakable
Put this alongside CrowdStrike’s 21st field and Google’s blank field, and you get three of the largest outages of the past eighteen months, all with the same shape:
A data or configuration change, generated or replicated automatically, reached everywhere at once, and hit a component that didn’t handle unexpected input gracefully.
So, once more for the people at the back:
- Treat generated configuration like a code release. Validate it, canary it, and stage it.
- Make limits soft where you can. Log, alert and drop the excess. Don’t crash.
- Keep a known-good version ready and make rolling back to it a single action.
- Don’t jump to “it’s an attack”. Keep both possibilities open until the evidence settles it.
