Reliability

The Bug That Waited Since July

Featured image: The Bug That Waited Since July. Microsoft's interim root cause: a latent defect, a routine failover, reduced capacity, and a cascade.

Updated 16 November with Microsoft’s interim root cause, and retitled to match (this post was first published as “Health Checks That Make Things Sick”). The original analysis below was based on the information available at the time, mainly the status updates Microsoft posted during the incident. Jump to the update.

If you tried to send an email or join a Teams call on Thursday afternoon, you probably noticed.

From the afternoon of Thursday 12 November (incident trackers log the start at about 16:17 UTC for Exchange Online), Microsoft 365 had a major outage. Exchange Online, Outlook, Teams and Copilot were affected for users in dozens of countries. Microsoft said early on that it was not a cyberattack. Service was restored in phases through Friday, and Microsoft has said a full post-incident review will follow.

At the time of writing that review isn’t out, so treat what follows as early analysis. It’s also the second big Exchange Online outage since the end of August, when a core authentication configuration problem knocked out email for much of a day. Microsoft’s updates during the incident described:

  • elevated load and servers becoming unexpectedly unavailable,
  • health checks marking servers as unavailable, which pushed their work onto the remaining servers, and
  • a recent network configuration change being rolled back as part of mitigation.

The Cascade

If that sounds familiar, it should. It’s the shape of a classic cascading (or metastable) failure:

  1. Something increases load or reduces capacity a little.
  2. Health checks see slow servers and take them out of rotation.
  3. The remaining servers get more load, slow down, and fail their health checks too.
  4. Retries from clients and other services add even more load.
  5. The system settles into a bad state that it can’t recover from on its own, even after the original trigger is gone.

We saw a cousin of this in AWS’s Network Load Balancer health checks last October, and in the OpenAI control plane the year before.

Defences Worth Having

  • Health checks that know the difference between “this server is broken” and “this whole fleet is overloaded”. Don’t evict half your capacity during a load spike.
  • Load shedding: reject some work early and cheaply, so you can serve the rest well.
  • Retry budgets and backoff with jitter, so clients don’t flatten you the moment you start recovering.
  • Plans for a controlled restart: how to bring a large fleet back without immediately overloading it again.

Update, 16 November: The Interim Root Cause

Microsoft has now published a preliminary post-incident report (in the Microsoft 365 admin centre under MO1550821 and EX1550543), and Charles Lamanna posted an abridged version. The Register has a good write-up with the timeline. The short version: three things lined up.

  1. A latent defect. A change made in July for the networking gateway servers that front Microsoft 365 caused lock contention under certain load, leaving servers unresponsive. It rolled out worldwide over several weeks and sat there for more than three months without anyone noticing. Load testing missed it because the test labs leaned on older hardware with fewer cores, and the problem is worse on newer machines with more cores.
  2. A load surge. About fifteen hours before the outage, Teams failed over onto the primary gateway that also fronts Exchange Online. That’s a normal operation, and by itself it wasn’t a problem.
  3. Reduced capacity. Nearly 14% of the gateway’s servers were out of service after an unrelated issue the day before. There was enough headroom, so that wasn’t a problem by itself either.

Together they pushed the gateway into the defect during the US peak. Servers stopped responding, traffic management moved users to the remaining healthy servers, and those then hit the same defect. In under two hours it had spread across the fleet, and at the worst point more than 80% of servers were unresponsive. The bulkheads that should have contained it to a region didn’t hold (Microsoft says due to a separate issue), which is how it went global.

Recovery was slow for two reasons. Servers that came back were swamped straight away by the waiting traffic and fell over again. And because the change was months old, the usual rollback, flighting and feature flags couldn’t undo it quickly. Engineers had to build an update that put the code path back to its previous known good state, then take large groups of servers out, patch them, restart them and bring them back. Teams, OneDrive and SharePoint could move to the other gateway, which is why they came back sooner. Exchange Online couldn’t.

By The Register’s timeline, monitoring flagged the problem at 16:03 UTC on Thursday, the mitigation was identified at 07:20 UTC on Friday, and services were confirmed healthy at 16:10 UTC.

What I Got Right, and What I Didn’t

My original post was written from what was available at the time: Microsoft’s status updates as the incident unfolded, which talked about elevated load, servers failing health checks and a configuration change being rolled back. The shape held: a classic cascade, load moving onto fewer and fewer servers, and a system that couldn’t recover without help. The controlled restart turned out to be the hard part.

What I didn’t have: this wasn’t health checks being too eager. The servers really were dead, from a defect that had been waiting since July. Moving their traffic to healthy servers is exactly what you want on a normal day. Here it spread the problem. The report also doesn’t mention retry storms, so treat that part of my list above as general advice, not a finding.

It adds a few lessons to the list:

  • Test on the hardware you run. If production has your newest, biggest machines, so should the load test.
  • Latent defects outlive your rollback window. Three months on, feature flags and flighting are no help. Know how you would roll forward in a hurry.
  • Test your bulkheads. Isolation you haven’t proven is only a diagram.
  • Failover moves load. Size your headroom for a failover and degraded capacity at the same time, because one day you’ll get both.

Microsoft says the final review is due later this week. I’ll update this post again when it’s out.

More Information

Leave a Reply