Personal

Blue Friday

A week ago today, on 19 July, a CrowdStrike Falcon content update for Windows caused systems around the world to crash into a boot loop. Airlines, hospitals, banks, broadcasters and retailers were all hit. Microsoft estimated that around 8.5 million Windows devices were affected. Mac and Linux machines were not.

Today is also SysAdmin Day, the last Friday in July. I don’t think the timing could be more appropriate.

What the Week Looked Like

For a lot of people in my line of work, it looked like this: a bridge call that started before dawn and didn’t really end. Thousands of machines needing hands-on recovery. And BitLocker recovery keys, which turned out to be the hidden pain point. If the key escrow sits on a server that is also blue-screening, you have a chicken-and-egg problem. Microsoft and CrowdStrike published recovery tools and guidance quickly, and CrowdStrike reported that more than 97% of Windows sensors were back online by 25 July.

CrowdStrike’s preliminary review points to a defect in a content update, the kind of rapid configuration change that is pushed often and was not treated with the same rigour as a software release. I’ll write more when the full root cause analysis is published.

To the People Who Fixed It

In 2020 I wrote a SysAdmin Day post called “Silent Running”, about the invisible work that keeps everything running. This week that work was suddenly very visible. People walked data-centre aisles with USB sticks. Service desks read out recovery keys one by one. Engineers in every time zone handed over to each other on calls. To all of them, and to my own colleagues: thank you.

Look after yourselves this weekend. The backlog will still be there on Monday.

More Information