Marriott Takes Disaster Recovery, Virtualization Underground

Hurricane Katrina and the September 11 attack on the World Trade Center caused scores of companies to reconsider their disaster recovery and business continuity plans, whether they were affected by those catastrophic events or not.

Marriott International was no exception. Marriott's outsourced disaster recovery program depended on a third party company that might require up to 24 hours to restore business critical systems. Plus, Marriott might need to compete with others for space at the recovery facility.

That will change dramatically early next year, when Marriott completes a new "in-sourced" Recovery and Development Center (RDC) in 12,500 square feet of space leased on a long-term basis from Iron Mountain. Marriott anticipates it will slash recovery time for business systems to within four hours, a standard that many other companies also have been trying to achieve.

But, what sets Marriott's RDC apart from the typical disaster recovery facility is its location -- 220 feet underground in a highly-secure and naturally-cooled former limestone mine located about an hour's drive north of Pittsburgh, in the US. When the RDC goes live, Marriott will be the largest private customer operating a data center in Iron Mountain's 145-acre "underground city," which has its own fire company, water treatment plant and 24-hour security and maintenance force to serve the 2,700 people working there from 18 companies and various government entities.

"The RDC demonstrates our dedication to developing innovative technology solutions for our business and the environment. The underground facility will enable cost-effective operations and is environmentally friendly, positioning us to lower our energy consumption," says Carl Wilson, executive vice president and CIO at Marriott.

Marriott's decision to shift from an outsourced cold site, which required a formal declaration process to activate, to an in-sourced hot/warm site, with dedicated space and infrastructure, falls in line with a trend that analysts have been seeing with increasing regularity.

John Morency, a research director at Gartner, says the need for quicker response and recovery times is driving many companies to weigh the cost of their existing contracts with external providers against the expense of doing the work internally. Plus, many IT staffs want added flexibility to manage and test their disaster recovery processes, rather than being subjected to the rigid time frames of many third-party agreements.

Virtualization's Big Role

Virtualization is helping to drive those trends because it affords a cost-effective way to create an in-house test environment, Morency says. Virtualization also aids with the actual movement of workloads from one set of hardware to another set of hardware.

Dan Blanchard, VP of enterprise operations at Marriott, says the company's strategy has been to use virtual servers wherever possible, which in turn, makes the recovery process quicker, easier and more reliable in that part of the IT environment. Because workloads are often shifted, moving them to a disaster recovery environment is just one more step, he says.

Another key advantage: potential errors or problems have likely been spotted and eliminated through ongoing testing, Blanchard says. Applications running on virtual servers aren't validated only on a quarterly basis as part of the disaster recovery test process; they're tested as part of normal operations, Blanchard says.

Page Break

But, Marriott hasn't been able to shift all of its applications to its VMware technology. Some vendors have refused to support their software if it runs in a virtual environment, and others haven't figured out how to charge for licenses, Blanchard says. So, like many companies, Marriott must maintain more traditional processes and procedures for the recovery of some of its business systems.

If Marriott's use of virtualization technology isn't especially unique, its selection of a physically inaccessible bunker for disaster recovery is, says Laura DuBois, an analyst at IDC (a sister company to CXO Media.) That decision would be more common for a financial services firm or a company based in Europe, where the level of security tends to be heightened over threats of terrorism and bombs, DuBois says.

Economic and Environmental Gains

Marriott, however, reasoned that the decision would make sense from both an economic and an environmental standpoint. The company calculated that the 10-year cost of co-locating a new data center at Iron Mountain's underground facility would be cost neutral compared to its existing agreement for disaster recovery, according to a spokesperson. Plus, the opportunity to improve energy efficiency would bring significant savings and help the company to achieve its environmental goals.

Blanchard says the key differentiator that pushed the Iron Mountain facility ahead of two other finalists was the environmental benefit. The top criteria to select the site had been mileage (because the company wanted IT staffers to be able to reach the facility without need of an airplane,) security and the corporate philosophy to "conserve and preserve," he says.

One of the chief costs associated with a data center is the power to keep the computers cool. Because the ambient temperature in the mine is 55 to 60 degrees Fahrenheit, Marriott will be able to reduce its energy consumption, as well as lower operating costs. The company also plans to take advantage of the mine's underground water supply to cool its equipment via chilling towers.

Marriott had hoped to pursue the Leadership in Energy and Environmental Design (LEED) designation from the US Green Building Council. But, there is currently no process to certify a subterranean environment, according to Charles Doughty, VP of "The Underground" at Iron Mountain. But, the mere reuse of an existing facility reduces the environmental impact that new construction would have brought.

The dual nature of Marriott's RDC-as not only a disaster recovery facility, but also a second data center-provides an added boon. The development, quality assurance and test systems in the underground facility will be active on a daily basis, until they're needed in an emergency recovery situation.

"The reason we did that is it saves us a lot of money. We're dually using those systems," says Blanchard. "That prevents us from having to go out and purchase and power a set of infrastructure that does nothing but sit there and wait for a disaster."

The systems will be identical to those in the production environment, so staffers won't need to brush up their skills on different technology in a disaster scenario. Virtual servers will ease the movement of some workloads from one set of hardware to another, even possibly from servers at the primary data center to servers at the new RDC, Blanchard says.

Initial plans call for two Marriott employees to work on provisioning and maintenance work at the underground site, with the potential to add more staff and space, if necessary, to accommodate business growth. Marriott has some 115,000 rooms in the pipeline over the next several years, according to a company spokesperson.

"Building this second data center," says Blanchard, "we position ourselves to be able to accommodate that kind of [business] growth."