Telstra has identified a “software fault” as the cause of a significant disruption to the telco’s 4G voice and services.
The outage began at around 10am yesterday. Telstra said that by 2.55pm it had returned mobile voice and data services to normal levels.
The fault “triggered multiple elements across the network to fail,” the telco said.
The 4G outage caused a large number of customers to drop back to 3G services — overwhelming the capacity of Telstra’s 3G network.
“The network is designed to switch onto standby hardware which it did,” a spokesperson for the company said. “Following the failover however, a further fault caused an interruption which impacted 4G connections. There is redundancy built into these systems but this did not operate as intended.”
By 11.40am yesterday, Telstra had isolated the affected infrastructure and begun the restoration of services.
“It took about two hours for full service to be restored after that, although our team was working longer with some government and enterprise customers to help them with their systems recovery,” the spokesperson said.
“We are still investigating the root cause of the software fault. We are working closely with our technology vendors on the specific element of software which triggered the issue.”
Telstra said its teams “worked around the clock” to restore services and to analyse the redundancy failure.
“We will continue to closely monitor network stability and performance,” the Telstra spokesperson said.
The outage is the latest in what appears to be a new horror run for the telco, following problems with 4G voice services at the start of the month and a major disruption to the Triple Zero emergency service.
Two years ago the telco suffered a string of significant outages including a major network meltdown that was attributed to human error.
The telco said a key cause of those problems was its network being overwhelmed as massive numbers of devices simultaneously attempted to re-establish a connection following an initial outage.
Following a review of its network Telstra CEO Andy Penn pledged to spend $250 million on improvements, including $50 million on boosting the mobile network’s capacity to handle re-registration of devices and on real-time monitoring.
Earlier this month Telstra CEO Andy Penn warned of investors of a “difficult trading period ahead”. He attributed it to the “challenging dynamics” created by the roll out of the National Broadband Network and increased competition in the mobile space.
The CEO said that over the last 12 months Australia had “moved from three big players in mobile and fixed to a situation today where we face a fourth network operator entrant in mobile” — TPG, which is preparing to begin customer trials later this year — “an increasing number of MVNOs and more than 170 resellers of fixed.”