TasWater archives to deal with data deluge
- 23 September, 2014 13:04
TasWater ICT manager David Byrne.
Tasmanian bulk water authority TasWater is implementing tiered storage to better manage its growing data volumes.
Much of the organisation's data comprises standard documents such as PDFs but there's also a significant amount of spatial data, TasWater ICT manager David Byrne said.
In 2009 TasWater's predecessor organisation Onstream had around 4.4 terabytes of data. It's now up to 60-70TB of data including operating systems.
"We've also been asked to help come up with a storage solution for about 12 terabytes of video data," Byrne said. The video data is generated by inspections of underground assets.
The water authority has recently implemented archiving, using CommVault to shift data that hasn't been touched for a number of years off disk and on to tape.
"We still have it but we're pushing it to tape to try to ease up some of the burden on our disk infrastructure," TasWater ICT manager David Byrne said.
"We've also started a project to do email archiving — again, keeping copies of all emails sent and received internally and externally, but after x amount of time we'll push it off to lower cost disk and again after some further time we'll archive it off to tape and delete it from disk."
A Hitachi tiered storage system will shift data up and down the different layers depending on how hot the data.
"We've got three levels of storage and the idea is by using CommVault, which we're also housing on the same storage, is that we dedupe what we can, particularly stuff that's not going to be accessed fairly often or at all."
TasWater, previously called Onstream, had its origin in a shared services organisation established to serve Tasmania's three former bulk water authorities, which delivered water services to councils.
That shared services model existed until last year when the four separate corporations were merged.
"Many of the councils realised the model that we had in place could deliver even better economy if we went from four corporations to one. We went from the four constituent water corporations, including Onstream, to one corporation on 1 July last year."
Managing the shift from four organisations to a single corporation was not without its challenges for the IT team, Byrne said.
"The beauty of the model that we built back in 2009 [when Onstream was created] was that we actually had multiple domains. We had a common domain called Resource that everybody shared and trusted all of the client domains, including our own, so that we could actually add and remove domains if the company got larger or smaller.
"It turned out it got smaller, so what we were able to do was unpick that. What we didn't quite factor in was a new domain called TasWater, and now we're porting everybody across to that.
"We've had quite a few challenges around migrating people and their computers, data and mail files from their original parent company domain to the new one."
With the amalgamation of the authorities, Byrne was faced with multiple storage technologies employed across the formerly separate organisations.
To keep costs down, TasWater was initially required to largely work with whatever the constituent organisations possessed in terms of IT assets.
"On day one, we actually had a mixture of technologies for backup/data protection," Byrne said.
Software for data protection included products from Vizioncore (later acquired by Quest Software, in turn acquired by Dell in 2012) — vFoglight, vReplicator and vRanger — ARCserve for tape backups, and Double-Take for managing block-level replication on databases.
"Between all of those, if one of them fails it had the potential to have a ripple effect on the others, and there were four different management interfaces to figure out how the backups were going every night," Byrne said. "It was a bloody nightmare."
TasWater went to market in 2011 to find a single pane of glass to manage it all. "I wanted to have confidence that I would get a report every morning telling me that the backups had worked successfully — so looking at a report full of green lines saying 'this all worked'."
The organisation tapped CommVault for the job and then implemented rapidly, Byrne said.
"It was a rapid implementation because we had to relocate in September 2011 — for us it was necessary to have everything backed up to be comfortable and confident that we had all the data safe and able to be recovered.
"When we put everything on a truck or two trucks, we wanted to make damn sure that if a truck fell over I could recover. Lucky for us we didn't need it, but I felt confident that if we had to ship in a whole bunch bare metal, we'd have been able to get it all up and running again."
TasWater recently purchased more CommVault licences to allow it to index and archive 30 million objects.
Byrne is also in the early stages of looking at a virtual data centre setup for the organisation, which will involve relocating one of TasWater's primary server rooms to a third party data centre.
"Our two primary server rooms are 200 kilometres apart, so the ability to do active-active data centre replication is just not there," TasWater ICT manager David Byrne said.
Having them that far apart is a "killer" because of the latency, Byrne said.
"What we're trying to achieve is actually having one virtual data centre across two physically separate sites, which is good for load balancing and good for disaster protection. And on top of all that, CommVault in the mix to back it up and archive it as required."