Last Wednesday morning Australia woke up to one of its biggest telcos simply… gone. From around 4am AEDT on 8 November, Optus mobile, home internet and landlines stopped working for more than 10 million people and around 400,000 businesses, for the best part of 12 hours. Trains in Melbourne stopped. Businesses couldn’t take EFTPOS payments. Most worryingly, some customers couldn’t reach 000.
I grew up in Townsville and spent a few years inside a mobile telco in the mid-2000s, so this one landed close to home.
What Happened
On Monday Optus published an initial explanation. After a “routine software upgrade”, the network received changes to routing information from an international peering network. Those changes spread through multiple layers of the network and exceeded preset safety limits on key routers. The routers then did what they were designed to do: they protected themselves by disconnecting from the IP core. Because so many of them did it at once, the core effectively shut down, and engineers had to physically reconnect or reboot equipment to bring it back.
So the protective mechanism worked perfectly, router by router, and the network as a whole failed. That’s a pattern I see again and again in complex outage recovery. Every individual component behaves correctly, and the system still falls over.
What I’d Be Asking
- Blast radius: Why could one upstream routing change reach every key router in the core? What stops propagation partway?
- Out-of-band access: If you have to drive to a site to reboot a box, recovery is measured in hours, not minutes.
- Diagnosis under pressure: Reports say engineers chased several theories in the first few hours. That’s normal. Having pre-agreed decision points for “stop diagnosing, start restoring” helps.
- Communication: For many customers, the silence was as frustrating as the outage itself. Tell people what you know, what you don’t know, and when you’ll next update them. I wrote last year about not staying silent. That applies to organisations too.
Back in 2020 I wrote about ISP bandwidth in an at-home world. This week showed how completely our homes, businesses and emergency services now depend on a handful of networks.
