Infrastructure

Same Day, Different Cloud

20 October is turning into a significant date in cloud history.

A year ago it was AWS us-east-1. This year, from around 06:08 UTC, a power event in zone europe-west2-b of Google Cloud’s London region disrupted dozens of Google Cloud products. Google’s status page listed 38 affected products. In the UK, customers of banks including Revolut, Lloyds and HSBC reported problems with apps and payments during the morning. Royal Mail, Rightmove, DPD and parts of the courts’ video system were also affected. Google’s incident record shows service restored by 14:00 UTC the same day, about eight hours after it began.

One Zone, Lots of Banks

Like AWS’s cooling failure in May, this started with something physical in a single zone. And again, the impact reached well beyond that zone. That’s because real architectures concentrate more on one zone, one region or one provider than their diagrams suggest.

For UK and EU financial services, this is exactly the scenario DORA and the UK’s operational resilience rules were written for:

  • Important business services have to stay within impact tolerances, regardless of whose infrastructure fails.
  • ICT third-party risk includes concentration: how many critical services depend on the same provider, region or zone.
  • Scenario testing should include “our cloud region loses a zone at 6am on a Tuesday”.

What I’d Ask on Monday

  1. Which of our important business services failed or degraded this morning, and did any go beyond its impact tolerance?
  2. Which of them have a realistic path to another zone or region, and which only have one on paper?
  3. Are we over-concentrated on one region in a country that only has one or two?
  4. How quickly did we tell our customers something was wrong, and did we tell them before social media did?

Two years in a row, the same date, two different clouds. I don’t believe in omens, but I’d book a resilience test for next 20 October, just in case. ?

More Information

Leave a Reply