Sometimes the root cause isn’t a race condition or a null pointer. Sometimes the room just gets too hot.
At around 5:25pm PDT on Thursday 7 May, multiple cooling units failed in an AWS data hall in availability zone use1-az4, in the US-EAST-1 region. Racks shut down to protect themselves, and EC2 instances and EBS volumes on that hardware lost power. AWS advised customers to use other availability zones while it brought extra cooling capacity online. Cooling wasn’t back to normal until Friday afternoon, more than 20 hours later. The Stack noted this was the fourth incident AWS had recorded for US-EAST-1 in 2026.
Coinbase halted trading for hours. FanDuel and others were also affected.
“But We’re Multi-AZ”
Many organisations hit last week were multi-AZ, at least on paper. The problems were in the details:
- One critical component, the system everything else waits for, was pinned to one zone.
- Failover hadn’t been exercised under real load, so it was slower or messier than expected.
- Some “global” services and control planes have dependencies routed through us-east-1 that customers can’t see.
Availability zones are a promise that one zone’s physical failure won’t take out the others. It’s a good promise, and AWS largely kept it. But it only helps if your architecture uses it, all the way down.
The Physical Layer Never Went Away
Over the past couple of years we’ve had a substation fire close Heathrow, an entire peninsula lose power, and now a cooling failure. As AI hardware packs more heat into every rack, power and cooling may become a bigger part of the reliability story, not a smaller one.
Some questions worth asking:
- Which of your components would stop everything if one zone disappeared? Find them and fix them first.
- When did you last deliberately fail an AZ in production, or something close to it?
- Do your providers’ status pages and contracts tell you enough about physical-layer risk in the facilities you depend on?
More Information
Postscript: Coinbase’s postmortem, published in June, found that architectural dependencies in its own systems, including a matching engine tightly coupled to the affected zone, turned a localised AWS failure into a multi-hour outage.
