The Commonwealth Bank has completed a mammoth overhaul of one of its key data centres in Sydney, delivering a boost to security, resilience and power efficiency, and dramatically extending the facility's projected lifespan.
The modernisation project for the 27-year-old data centre began in 2012 and involved years of design work and business case development before a contract was signed in January 2015.
Despite the scope of the works, which involved creating two new data halls and consolidating on a single floor of the CBA-owned, HP Enterprise-operated 19,000m2 facility, the data centre remained live during the project.
“It had just got to end of life so we had to make an investment decision around it,” the bank’s CIO, David Whiteing, says of the data centre project.
“At the time when we were making the decision didn’t have a strong sense of where the regulator [the Australian Prudential Regulation Authority (APRA)] would be, particularly around cloud and what we could put into the cloud, so we made the commitment that everything that we do basically has to be modernised.”
A more “coherent conversation” around cloud is emerging, Whiteing says. However, he adds: “I can’t see a world where in the next five to 10 years we still wouldn’t need to have our own data centre capability.”
A hybrid approach combining the use of public cloud with internal cloudlike capabilities “is absolutely part of the reality of where we need to go,” he says.
Overall the project took 220,000 person hours and included the replacement of electrical, mechanical, fire and security systems, increasing the facility’s projected lifespan by 15-20 years.
The decision to revitalise its existing facility instead of moving to a new data centre is believed to have saved the bank hundreds of millions of dollars.
To keep the facility operating and secure during the work, the live environments were secured and new entrances created for construction contractors.
Data halls on other floors are being progressively cleared out, offering opportunities for future growth if necessary.
Whiteing says that the importance of the process stretches beyond the immediate impacts on the data centre’s resilience and efficiency.
“You get other efficiencies around change”, the CIO says. “Change just becomes easier, quicker as you modernise things.”
The data centre environment had a complex and lengthy technical history. “You needed an event like this to reset it,” Whiteing says.
“More and more of our customers are demanding that we’re always available; not just available other than maintenance windows,” the CIO adds. “You have to modernise to do that.”
Behind the scenes
The project involved extensive full integrated services testing, including putting on 1.1 megawatts of IT load and undertaking 110 separate failure scenarios across two weeks.
The bank has put in a highly available modular electrical design, moving from 250kw modules to 14kw modules that are hot swappable in production. The design is based on four separate power systems and can withstand up to 10 component failures per module.
Power efficiency has been increased, with the data centre moving from 1.8 to 1.4 PUE.
The cooling and backup generator systems have been augmented. Instead of using high velocity, low volume forced cooling the data centre employs a displacement system that uses slowly spinning, large fans that flood halls with cool air.
Cold air comes into the front of a rack, runs over the processors and is ejected out the back, then travels up to the ceiling space of the data hall and into a service corridor.
The modular AC system can withstand two major component failures per module before switching to a backup service.
The data centre’s backup generator system has gone from N+1 to N+2.
The bank also implemented a new biometric security system for the facility that relies on iris recognition. According to the bank, the system offers the highest level of authentication currently available on the market, with 200 points of identification compared to around 60 for a palm reader.
Gallery
Inside one of the new data halls:
The data centre has four separate power systems:
Racks have fully monitored power rails:
Revamping the data centre's cooling included ripping out the ceiling:
This data hall remained live during the project:
Storage — the bank needs a lot of it, obviously:
A CBA data warehouse:
As you would expect, the data centre is packed full of a range of hardware from a range of vendors, running everything from data analytics software to CBA's network of ATMs:
Security is courtesy of iris scanners:
The project included putting in new cooling and fire suppression systems:
