Noel Leslie's voice cracked momentarily as he recounted the night last year when his team at SBS rescued the broadcast of the 2018 FIFA World Cup from Optus Sports' streaming service meltdown.
"I'm incredibly proud of the team. I get emotional just thinking about it... it was a massive, massive thing we did," the chief technology officer said.
Even before the emergency agreement on day three of the soccer competition that saw SBS broadcast all 57 remaining matches, the challenge was considerable. Fans would not forgive any snafus.
"It's a massive event that people get emotionally attached to," Leslie told the Amazon Web Services summit audience in Sydney.
But thanks to months of preparation, dedicated staff and partners, and experience earlier in the year handling major events like Eurovision, SBS was ready to rise to the occasion.
At the time the broadcaster was halfway into its three year 'digital acceleration' plan, which aims to move it from a linear broadcast organisation to one that serves up content on demand.
A key part of that plan was a move to a business-wide media asset management system, which brought together five different content systems each with different storage, tools and processes.
"It means we've improved collaboration and speed of access to content. It used to take us, believe it or not, 24 hours to get a piece of audio off TV onto radio, because of all the systems and it had to go through various transformations – now it's done instantaneously," Leslie said.
Given around 700 of SBS's 1000 staff use the new platform, there had been a considerable training and change management effort.
The broadcaster had also moved more of its infrastructure to the cloud, automated a number of workflows and adopted agile practices. It worked with a multitude of partners to realise the plan, including Adobe, Cisco, New Relic, Dalet, Dell EMC and AWS.
Leslie and his 40 strong IT-team adopted AWS cloud services such as EC2, Aurora DB, Elastic Search and AWS Elemental for video transcoding and encoding. Akamai was leveraged for the content delivery network, and New Relic for monitoring.
"Those partnerships give us access to expertise and resources we just don't have. And it's never just about the technology it's about the people. Live TV is inherently scary, you want to know you can pick up the phone to somebody somewhere who is going to make something happen pretty quickly, not rely on a ticket," Leslie said.
SBS had been readying itself for its role in the World Cup with some major broadcast events earlier in the year including the Eurovision Song Contest and the first series of The Handmaid's Tale, which gave it experience in handling huge streaming audiences.
"It gave us some indication of what things might be like. These large-scale events created opportunities to prepare and test infrastructure," Leslie said.
As well as the technical elements – including appliances that burst over to the cloud and stringent load testing – the events, particularly Eurovision and Sydney Mardi Gras coverage meant the broadcaster was used to dealing with viewers late at night and in the early hours.
"Sometimes people attempted to fire off a tweet – 'this isn't working' – now the biggest surprise to them at 4am in the morning is when they get a polite tweet back from us – ‘Have you tried this?'" Leslie said.
Nobody was quite sure how large Australian audiences for the World Cup might get.
"Some people said it would be big with a capital B, but that didn't help," Leslie said.
"My mother used to say I was a born worrier, and it's probably very true. It's served me quite well...I was passionate, or petrified, that scalability was going to be a big issue on this work," he added.
Come June, the testing and preparation for SBS' ninth World Cup (and Leslie's second with the broadcaster) was paying off.
Things weren't going as smoothly at Optus Sport.
Just days into the competition, subscribers of Optus Sport (which had bought the rights to air many of the major matches) faced error messages, connection issues and drop outs as they attempted to stream games.
Fans soon had a name and hashtag for the failure: Floptus. A simulcast arrangement was quickly forged.
"It was only at 5pm that we agreed that we'd take the extra services from Optus. So we then had a few hours to work out how we were not only going to do what we normally did that night, but how we would cater for the 1am game and the 4am game having nobody scheduled to be there," Leslie said.
The simulcast workaround was initially planned to last 48 hours, with Optus CEO Allen Lew releasing a statement to say the telco had the “capacity and capability” to meet the needs of fans, and had dealt with the failure in a “critical part of our content delivery network”.
The simulcast was soon extended to the end of the competition.
"For those kind of moments all you really can do is buy a lot more pizza and beer," Leslie says.
SBS aired all 57 remaining games on TV and online and on radio (often in up to five languages). At its peak, 280,000 users streamed the Australia vs Denmark match.
The broadcaster served 7.8 million streams on radio and digital, reaching 9.1 million Australians and "didn't lose a second" Leslie says.
The experience was an emotional but rewarding one, Leslie said.
"And just when you thought it was calming down, the FIFA Women's World Cup starts in five weeks’ time. So we're going to get to do it all again!" he added.