If you tried to send an email or join a Teams call on Thursday afternoon, you probably noticed.
From around 12:30pm Eastern (17:30 UTC) on Thursday 12 November, Microsoft 365 had a major outage. Exchange Online, Outlook, Teams and Copilot were affected for users in dozens of countries. Microsoft said early on that it was not a cyberattack. Service was restored in phases through Friday, and Microsoft has said a full post-incident review will follow.
At the time of writing that review isn’t out, so treat what follows as early analysis. It’s also the second big Exchange Online outage since the end of August, when a core authentication configuration problem knocked out email for much of a day. Microsoft’s updates during the incident described:
- elevated load and servers becoming unexpectedly unavailable,
- health checks marking servers as unavailable, which pushed their work onto the remaining servers, and
- a configuration change being rolled back as part of mitigation.
The Cascade
If that sounds familiar, it should. It’s the shape of a classic cascading (or metastable) failure:
- Something increases load or reduces capacity a little.
- Health checks see slow servers and take them out of rotation.
- The remaining servers get more load, slow down, and fail their health checks too.
- Retries from clients and other services add even more load.
- The system settles into a bad state that it can’t recover from on its own, even after the original trigger is gone.
We saw a cousin of this in AWS’s Network Load Balancer health checks last October, and in the OpenAI control plane the year before.
Defences Worth Having
- Health checks that know the difference between “this server is broken” and “this whole fleet is overloaded”. Don’t evict half your capacity during a load spike.
- Load shedding: reject some work early and cheaply, so you can serve the rest well.
- Retry budgets and backoff with jitter, so clients don’t flatten you the moment you start recovering.
- Plans for a controlled restart: how to bring a large fleet back without immediately overloading it again.
I’ll update this post when Microsoft publishes the review.
