Hurricane Katrina made landfall last Aug. 29, battering Gulf Coast areas in Mississippi, Alabama and Louisiana, killing more than 1,800 people and causing an estimated US$75 billion in damage. New Orleans took the brunt of the storm's fury -- once the levees gave way, 80 percent of the city flooded.
To find out how the city is rebuilding its telecom infrastructure and revamping its disaster-recovery plans, we sent a team to New Orleans to report on the efforts of service providers, city officials and enterprises to prepare for the next Katrina.
While signs of devastation are everywhere and the city certainly won't be back to its old self for years, there are many encouraging signs. When carrier upgrades are completed, New Orleans will enjoy the most advanced wired and wireless telecom infrastructure in the country. The city itself is determined to become a national model for connectivity, high-tech customer service and disaster readiness. And companies are shoring up their disaster-recovery plans based on lessons learned last year.
Mission critical is one of those terms that gets thrown around a lot, but what takes place at Lockheed Martin's Michoud Assembly Facility in New Orleans is mission critical -- the design and manufacture of external fuel tanks for NASA's Space Shuttle program.
Last August, engineers at Michoud had a pretty important project on their plates. "We had a shuttle flight with some debris falling off of it, and we were analyzing that," says Steve Stefancik, IS director at the company.
Then Katrina hit. While most employees evacuated, Stefancik kept a 37-man crew on site throughout the storm. The crew was able to keep the water pumps going until the last minute, and after the storm passed, Michoud was one of the only enterprises in New Orleans that didn't flood out.
Unfortunately, the roof of the building sustained damage, and 57 of the plant's 100 servers suffered hard drive damage. Lockheed Martin had replacement drives delivered to the plant by hitching a ride on a National Guard convoy, and Stefancik was able to bring the internal network up by Sept. 13.
But he still couldn't get connectivity to other sites around the country. "All of our communications going off-site went through a central switch down the road that flooded, and it took out all our wide-area network stuff," Stefancik says.
So Lockheed Martin arranged for another NASA site, the Ames Research Lab in California, to fly in via helicopter with satellite-based WAN connectivity. "It worked pretty well,'' Stefancik says, but the engineers in New Orleans needed to have some of their work processed at the Marshall Space Flight Center interstate, and the satellite link didn't provide enough bandwidth.
Because of the urgency of the project, Lockheed Martin ended up relocating the engineers and the data to the Marshall Space Flight Center.
No WAN
Despite all of the disaster-recovery planning that businesses in New Orleans did, no one was quite prepared for the total loss of WAN connectivity.
For example, Phelps Dunbar, a New Orleans law firm, thought it had a solid disaster-recovery plan. The company, which has seven sites in the United States and one in London, had just put the finishing touches on its MPLS network shortly before Katrina struck. "That gave us any-to-any connectivity, allowing us to relocate and restore critical systems in New Orleans to our other offices," said Chris Rigamer, director of technology
His disaster plan was to move the company's primary network and computing assets to its Jackson, Miss., office and recover from there. The Jackson computer room had been built three times larger than necessary so that servers and network equipment could be purchased and brought there if needed, duplicating the services in New Orleans.
And Rigamer specifically went out and added a second Internet link from a different ISP. Unfortunately, both ISPs went down when Katrina hit. "So we were doing a lot of stuff right and by the book, and we still got caught short,"he says.
Forced to improvise, he was able to take the link dedicated to the WestLaw legal research service and repurpose it for general Internet connectivity. "It was a tiny amount of bandwidth compared to what we needed, but it was a link," he says.
Since then, the company has added another Internet connection through the MPLS network.