Reliability

Twenty-One Fields

Following on from Blue Friday, CrowdStrike published its full root cause analysis on 6 August. It’s twelve pages and well worth reading. In short:

  • A new IPC Template Type in the Falcon sensor defined 21 input fields.
  • The code that called the Content Interpreter for those templates only supplied 20.
  • Earlier template instances used a wildcard for the 21st field, so the mismatch never showed up.
  • On 19 July, new template instances used a real value in the 21st field. The interpreter tried to read an input that wasn’t there, an out-of-bounds read, and Windows crashed.
  • The Content Validator had a logic error that let the faulty content through, and there was no runtime bounds check to catch it.

CrowdStrike says it has added bounds checking and input-count validation, more tests, and staged deployment of this type of content. It has also commissioned independent reviews of its code and processes.

What I Take From It

None of these causes is exotic. Each one is the kind of thing every engineering team has done at some point. The outage came from several of them lining up at once, in software running in the Windows kernel, pushed simultaneously to millions of machines.

  1. Configuration is code. If a “content” or “config” change can crash a system, it needs the same testing, validation and progressive rollout as a code change. This is the lesson I’d underline twice.
  2. Validate at every layer. The validator checked what it was told to check. The interpreter assumed the validator was right. Defence in depth applies to correctness as well as security.
  3. Stage everything. Canary first, then a ring, then the world. Rolling out to everyone at once is how a bug becomes a global outage.
  4. Recovery design counts too. How quickly can you get a machine back if it can’t boot? That question deserves engineering effort before the day you need the answer.

My employer, Kyndryl, published its own reflections on the incident, including how its teams used AI-driven visibility through Kyndryl Bridge to identify and prioritise impacted systems. Kyndryl has also said publicly that most of its affected customers were fully recovered within 24 hours.

More Information