New Orleans IT departments brace for Tropical Storm Gustav

As Hurricane Gustav approaches the US Gulf Coast, the IT lessons learned from the devastating Hurricanes Katrina and Rita that smashed New Orleans and other areas in 2005 are on the minds of many worried IT managers.

"We definitely have concerns about it, especially after what happened three years ago," said David Avgikos, president of Digimation, a 3-D digital animation software company, 10 miles west of New Orleans. "We don't have to be told twice."

In 2005, when the two hurricanes hit the city in quick succession, Digimation was knocked out of business for a week, and its electrical power and Web site weren't restored for two weeks. For Avgikos, those events were lessons learned.

Today, all of the company's approximately 50 PCs and servers are backed up to a main server, which is then backed up to a portable 1TB USB drive that goes out the door with the last employee who is evacuating in the event of an emergency, he said. That employee also takes an "emergency box" that includes additional DVD backups.

And the company's Web site, which was hosted in-house when Katrina and Rita struck, are now farmed out to a remote hosting company far from New Orleans so the Web site stays in operation even if the business is being affected by an emergency.

"Now, in the worst case scenario, wherever we locate to, we can go to the nearest Best Buy and get what we need to be up and running in a few hours," Avgikos said.

The company's e-mail server is remotely located, too, to ensure that e-mail access is available from wherever the company's 12 employees are in the event of an emergency.

Page Break

Loyola makes more preparations

Meanwhile, at Loyola University's New Orleans campus, a key new IT disaster recovery component has been added since the 2005 hurricanes. It is an intermediate site hosted at an out of state location set up to maintain communications and a simplified version of the school's Web site that's available with a simple DNS change.

"We had a very full-fledged disaster recovery plan prior to Katrina," said Bret Jacobs, executive director of IT for the school. But even so, what they learned from those earlier storms was that even more preparations were needed.

"Because of Katrina, we lost power in New Orleans for an extended period of time, and we realized that we wanted to keep that particular presence on the World Wide Web all the time." To do that, the school added the intermediate site for a third level of redundancy in an emergency.

That intermediate site complements on-site emergency generator power and air conditioning for the IT department, as well as complete off-site critical systems redundancy provided by disaster recovery vendor SunGard Availability Services, Jacobs said.

Since Katrina, the school has also moved its course management system online and has it hosted remotely so that students can continue their classes over the Internet in the event the campus is closed due to emergency. "What we're trying to do is make sure the educational process is not interrupted," he said. "We're not a distance learning campus per se, but we're using some of those technologies." With the approach of Tropical Storm Gustav, the university's IT department and its disaster recovery vendor have been put on alert mode, Jacobs said. Full system backups are being performed daily and are being shipped offsite as part of a standard IT plan.

"We want to have our reactions institutionalized because you never know what a storm will do," Jacobs said.

The school conducts a mock IT disaster recovery exercise annually with SunGard, bringing up the systems live at a remote facility in Chicago to ensure that they'll work if needed in an emergency, he said. "Your plans have to be flexible and you have to be able to adapt them to whatever is thrown at you."

Loyola also canceled classes Friday, saying Gustav "posed a significant threat to the city of New Orleans" and ordered all students and staff to evacuate. It also suspended all school operations tomorrow, according to a message posted on its Web site.

Page Break

Hospital takes steps to stay connected

At St. Tammany Parish Hospital, spokeswoman Melissa Hodgson said Katrina's fury prompted the 237-bed hospital to install a satellite communications system so that telephone and Internet connectivity could be better maintained in future emergencies.

Having communications systems knocked out for weeks or longer was one of the greatest challenges for recovery efforts after the 2005 storms, Hodgson said in an e-mail.

"This system is routinely tested to give us certainty that we will have open lines of communication should another Katrina-level disaster strike," Hodgson wrote. Data backups of critical systems have also been moved out-of-state in preparation for Gustav's arrival, she said.

The hospital's data center is in a bunker designed to withstand hurricane force winds and is located 25 feet above sea level to prevent flooding, Hodgson site. Off-site data recovery and redundancy services are provided by SunGard in the event that local facilities are damaged.

At Tidewater, a New Orleans company that provides support, assistance, boats and crews to oil and gas exploration and production companies in the Gulf and elsewhere, IT operations have undergone major improvements since Katrina.

"Having learned from Katrina and Rita, we now have a totally redundant IT system in Dallas, instead of in New Orleans," said Joe Bennett, an executive vice president and chief investor relations officer. "We can just flip a switch. We didn't have that redundant system back then."

'You have to be prepared for the worst'

At Tidewater, company officials are closely monitoring weather reports to see where Gustav is heading and to gauge its strength, Bennett said. The company has about 8,400 employees, most of whom work on ships. About 70 people work in the company's New Orleans headquarters, including about five IT workers. Tidewater also has offices in Houston and its ship operations base is in nearby Amelia, on the coast.

"We believe that we are ready" for the approaching storm, Bennett said. "We've had a lot of training and we meet daily ... with our key personnel. This one is still a few days away. You have time to plan but you don't know what exactly what you're planning for. Do you revert your phone services, etc., to Houston or to New Orleans? You have to stay kind of light on your feet when you're planning for a hurricane. We've been through this before."

"What we learned from Katrina and Rita is that you can't be too prepared," he said. "Will this be a false alarm? Possibly, but you have to be prepared for the worst. There's no reason to panic."