On Thursday 12 June, a large part of the internet had a bad afternoon. Google Cloud APIs returned errors around the world for several hours, and services built on them followed, including Spotify, Discord and parts of Cloudflare. Google published its incident report on 14 June. Like the best post-incident write-ups, it’s refreshingly direct.
What Happened
- On 29 May, new code was added to Service Control, the system that authorises Google Cloud API requests and enforces quota policies. The new code path had no proper error handling and wasn’t behind a feature flag. It sat there, untriggered.
- At about 10:45am PDT on 12 June, a policy change containing unintended blank fields was written to the regional Spanner tables that Service Control reads.
- Quota policy is global, so the change replicated worldwide within seconds.
- Every Service Control instance hit the new code path, hit a null pointer, and crashed into a restart loop.
- As a bonus, Google’s own Cloud Service Health status page was affected, so the first public update was delayed.
The incident lasted from 10:51 to 18:18 PT. Most regions recovered within a few hours. Some took longer because restarting Service Control put a heavy load on the systems it depends on.
The Familiar Shape
As an interesting aside, this is the second Google Cloud story on this blog with a blank value at its heart. The first was UniSuper last year. It’s also a cousin of CrowdStrike’s 21st field: code that’s fine until a particular piece of data arrives.
The lessons are almost a checklist by now:
- Feature-flag new code paths, especially in critical systems. A flag turns a global outage into a toggle.
- Validate data as carefully as code. Configuration and policy changes deserve staged rollout too.
- Fail open or fail safe on purpose. Decide in advance what a critical component should do with input it doesn’t understand. “Crash” should not be the default.
- Host your status page somewhere that won’t fail with you.
- Plan the restart. Systems that all restart at once can knock over their own dependencies.
