Disaster planning, mix-and-match style

Disaster planning traditionally focuses on three variables: data center replication, building design and backups. Analysts have maintained for years that the most common "disaster" is outright hardware failure because of faulty data center design, for instance, when the "emergency power off" button is hit, either accidentally or on purpose. Yet, for many enterprises throughout the US, the reality is that recovery plans should be customized for whichever type of major disaster is most likely to occur in any given area.

"There are really two kinds of disasters that can affect your data center," says Ken Brill, founder and executive director of The Uptime Institute. "Those that do not affect your data center directly but do affect your region -- when the region recovers, you will recover. Another is a disaster that affects your building directly; you won't recover until you recover the building. One of the most important decisions, but one that is often given little thought, is where to put the data center."

According to Brill, regional considerations such as city location, proximity to the ocean, whether the data center will be near a flood plain and even variables such as whether the data center is near an exterior wall, should all be considered carefully based on the region, not on broad computing guidelines.

"There's a wave of regional events around the world, which started with 9/11 and then the power outage during the summer of 2003 [on the US East Coast], Katrina, and flooding that we see going on around the world and terrorist events," says Roberta J. Witty, an analyst at Gartner, who says disaster recovery budgets should be 8 per cent to 9 per cent of total data center budgets. "Organizations are starting to understand that an event can happen completely outside of your control and you'd better be prepared for it."

To find out how different companies customize their plans, we spoke to data center administrators in several locations around the US, including Florida, Minnesota, California and New York.

Earthquakes

One of the primary concerns with earthquakes is prepping the building for both tremors and catastrophic events. Lucasfilm, located north of San Francisco in California, anchors each building to the ground -- including video production facilities and the main data center. The location is built on bedrock and is well above sea level. Kevin Clark, IT director at the data center, says these precautions were planned well ahead of the site's construction and were the primary methods they used for disaster planning.

"We have production facilities in Singapore and at Skywalker Ranch [the home base for George Lucas, located on the same campus as Lucasfilm], and we rely on a core infrastructure -- the data and applications we use at each site, which we could use to carry on with our work," says Clark. "The architecture of the building is designed to give a little if there is a tremor. We also have 1.5-megawatt generators on site, and run our entire data center on UPS. We're always trying to improve our disaster planning, but we also recognize that we can't sacrifice the work at hand for any long-term plans that may or may not be that effective in an earthquake."

Clark explained that the creative environment in which they work has to take priority over technical plans for disaster recovery, including the location of the data center itself. A more extreme disaster plan, such as locating the data center in a less earthquake-prone area, would hamper creativity when the throughput was not as high without the data center nearby.

The Uptime Institute's Brill says that a more catastrophic event such as an earthquake requires a plan for replicating data to a safe location, and because building codes are typically geared for earthquakes (as opposed to codes suited to other areas) the focus should be on contingency plans for data.

Page Break

Hurricanes

Disaster recovery plans for hurricanes usually differ from other disaster plans because there is a much longer period of time before you can physically access a data center again. Ben Weinberger, IT director at the Ruden McClosky law firm in Florida, says his company's disaster planning process is uniquely geared to the region. No data center in their multiple offices -- many of which are along the coast -- is positioned by exterior walls, and data centers are not situated on the lower floors (where flooding can occur) or on the top floors (where rain can seep in during a storm). Mission-critical data centers are situated off the coast, in Orlando and Chicago. The main replication site is in Chicago and would not be affected by a hurricane.

"We never put computers or servers near windows, and all servers are fully replicated now like they were in both Hurricane Wilma and Hurricane Katrina," says Weinberger. "If we had an imminent catastrophic event, we can switch over to Fort Lauderdale or Chicago using CA software replication with a few clicks. About 90 per cent of the time you can see a disaster coming and be preemptive."

Although replication is not a new concept -- many companies in New York have instituted a replication process since 9/11 -- it is becoming more common in areas where a different kind of disaster, such as a hurricane or tornado, is more of a concern than a terrorist attack. Weinberger said the costs are high for replication because it means running a secondary data center -- usually at a hosting provider -- but said the expense is worth it when data loss means a total long-term business outage.

Weinberger says one key problem with hurricanes, as opposed to another disaster, is the long-term power outage in the region. He says a data center could be left without any damage at all, but power might still be out for weeks or even months, which is why they replicate data to a facility in Chicago.

"A disaster has to be customized for the location and risks," says George Hamilton, an analyst at Yankee Group Research. "For example, little if any damage was done to the archived papers, tapes and backup materials stored in Iron Mountain's facility in New Orleans during Katrina. Pretty much everything was high and dry."

Despite that, it still took a long time for Iron Mountain to get access to the facility to recover everything. A lot of businesses had disaster recovery plans that they could not execute because they never planned for not being able to get to the protected storage facility, Hamilton said. "The decision on what to do is largely based on how likely it is that you'll face a natural disaster," he explained.

Terrorist attacks

Most companies have extensive plans in place for cyberattacks, but a physical attack against the US, such as what occurred on 9/11, requires a more specific plan. Martin Silverman, IT director at furniture distributor EvensonBest, has extensive plans that go beyond the typical methods used at other companies. He uses replication software -- again from CA -- from the company's headquarters to two different regional sites. The company runs replication services four times a day and verifies e-mail to use as documentation in case physical documents are destroyed. He also has a disaster plan in place (similar to a fire escape plan) so that employees know where and how they can work if the building is not available.

Page Break

Silverman suggests "employing an IP-based communications service that can forward trunk lines to a designated fail-over sites within one to two minutes automatically [if a catastrophe occurs]."

"Since 9/11, the rule of thumb is to move your backup center so it is geographically separate from the primary locale," says Charles King, an analyst at Pund-IT. "The companies that did relatively well on 9/11 in some cases had their backup centers 50 or 100 miles away. It was a short drive but still far enough away that they were able to roll over all their processes to the secondary site."

Long-term power outages

Unlike most other natural disasters, or a terrorist attack, a power outage is common across many regions and requires a more general plan: backup generators, UPS used for servers and storage, and a contingency plan to help workers get back online and become productive at a separate facility.

However, more extensive plans are required in highly populated regions such as California and the East Coast because of an increased likelihood of long-term, debilitating power outages from power shortages and high use in the area. Pund-IT's King says the strategy should include having on-premises backup replication for every server in the data center. Also required is a disaster recovery facility that's ready and available in case a power outage makes the primary data center unusable. This secondary site should be in place, available and equipped for the power outage, but it does not have to be a permanent site.

"In California about three or four years ago during the height of the Enron scamming of the power grid, Northern California went through a series of potentially very serious brownouts and blackouts because there simply wasn't enough power to go around," says King. "The availability of power as data centers increasingly consume more power is an issue that I think is on everyone's mind. It's one of those things that is pushing the adoption of green IT and server and storage consolidation solutions as well."

Tornadoes

Most of the experts agreed that a tornado does not warrant as specific a plan as earthquakes, hurricanes or a terrorist attack. The reason: A tornado is often less destructive for as large of an area as an earthquake, and does not typically lead to incapacitating the data center as much as a widespread hurricane. Still, Brill noted that the data center location is still critical: not near an exterior wall or window bank, data replication to a separate facility, meeting or exceeding building codes in regions such as the Midwest where construction is more suited to keep the building safe from tornado activity.

"It is relatively cheap to make sure your building is hardened from a tornado, but much more expensive to make sure it can withstand one and you can continue operations," he says.

Customized plans for the enterprise, critical as they are, are often the most difficult to create. Many DR vendors provide "stock" services that aren't necessarily based on specific disasters. Experts recommend hiring a local consultant who specializes in disaster planning for your area, talking to building designers about architectural guidelines for the data center and replicating data off-site at a distance.