On Monday 20 October, a large part of the internet had a very bad day. AWS’s US-EAST-1 (Northern Virginia) region had a disruption lasting around 15 hours, and with it went countless apps and services: banking, gaming, messaging, smart doorbells and more. AWS published its summary on 23 October. It’s one of the most detailed public write-ups AWS has released.
What Happened
- DynamoDB uses an automated system to manage DNS for its endpoints. One component (the DNS Planner) produces plans, and others (DNS Enactors) apply them.
- A latent race condition between two Enactors meant one of them applied an old plan just as a clean-up process deleted it. The result was an empty DNS record for
dynamodb.us-east-1.amazonaws.com, and the automation couldn’t repair it. - From 11:48pm PDT on 19 October, anything trying to open a new connection to DynamoDB in the region failed. Engineers found the DNS state as the source within the hour and fixed it by around 2:25am.
- But DynamoDB sits underneath many other AWS systems. EC2’s internal instance-management systems and Network Load Balancer health checks were knocked into states that took until the afternoon to work through.
AWS has disabled the DNS Planner and Enactor automation worldwide while it fixes the race condition and adds protections.
Three Lessons
- Find your hidden region dependencies. Plenty of “multi-region” designs still route sign-in, configuration or a global service through us-east-1. Draw the map before the outage draws it for you.
- The recovery tail is the real outage. The trigger was fixed in a few hours. Clearing the backlog, retries and throttles took far longer. Plan, and test, how your systems behave when a dependency comes back.
- Automation needs guardrails too. Automation that can delete a record should also be able to check it isn’t deleting the last good one.
Same day, my employer released its 2025 Readiness Report, covering 3,700 senior leaders across 21 countries. It wasn’t planned that way, but I couldn’t have asked for a better illustration of why readiness matters.
