Infrastructure

Two Geographies, One Delete

Another Australian story this month, and another one I’ve been explaining to friends back home.

From early May, members of UniSuper, a superannuation fund with around 647,000 members, couldn’t access their accounts for about two weeks. On 8 May, UniSuper’s CEO and Google Cloud CEO Thomas Kurian issued a joint statement, which is rare in itself. It explained that an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services had led to the deletion of UniSuper’s Private Cloud subscription.

UniSuper had duplication across two geographies to protect against outages and loss. The deletion removed both.

Services were restored by mid-May, helped by backups UniSuper held separately. On 24 May Google published more detail. A provisioning tool had been run with a parameter left blank. That caused a default fixed term to be applied, and the environment was deleted automatically when the term ended. Google says it has since changed the tooling and processes involved.

The Lesson I’ll Keep Repeating

Redundancy and backup solve different problems:

  • Redundancy protects you from things breaking: hardware, zones, regions.
  • Backups protect you from things being changed or removed: deletion, corruption, ransomware, a mistake in a control plane.

If both your copies sit under the same control plane, the same account and the same set of permissions, then a single delete — accidental or malicious — reaches both. Geography doesn’t help.

Some practical checks:

  1. Do you hold at least one copy of critical data outside your primary provider’s control plane?
  2. Are backups immutable or logically air-gapped?
  3. Have you actually restored from them, at scale, within your recovery time objective?
  4. Do your provider contracts say what happens if the provider itself deletes your environment?

Credit where it’s due: both organisations communicated clearly once they knew what had happened, and UniSuper’s separate backups made recovery possible.

More Information