SAN MATEO (07/05/2000) - Frank Woods remembers his exact whereabouts when he found out just how painful and costly the words "I Love You" could be. It was May 4, recalls Wood, COO of the New Mexico State Highway Department. He was sitting in a meeting in Albuquerque when he received a priority call from his Santa Fe office. The news struck him like a freight train: Massive amounts of IT infrastructure in his department had been infected by the mercurial e-mail called "I Love You" that was spreading a file-and JPEG-deleting worm and wreaking global havoc.
"We use a lot of JPEG files, digital cameras, and scanned photographs in our engineering design process," Wood says. "Our hardest-hit area was engineering.
We believe there may have been 50,000 to 60,000 files affected."
The task of cleaning up the infected machines was hampered by a tricky wild card: The department was in the midst of integrating Microsoft Corp. Exchange and Microsoft Outlook as its single e-mail program and installing a complete Tivoli framework infrastructure. Making matters worse, Wood says, his organization did not have an emergency plan in place to deal with the threat.
The first order of damage control was to shut down every piece of equipment throughout the legacy system. It lay dormant for about two days.
"It's safe to say everything from design processes to having systems and communications to our consultants and contractors -- every part of our technology was shut down. We literally went back to paper and pencils for many of our processes," Wood says.
The second step was to set up an IT crisis management team; individuals from Wood's staff, Tivoli Systems Inc., Compaq Computer Corp., Fiori Industries, and the Foundation Consulting Group issued hourly reports.
Less than 24 hours after the initial lockdown, individual parts of the system were turned back on, only to be aborted after it be-came apparent the worm continued to propagate through outside sources. Because the Tivoli environment was not yet fully implemented, any remote user not accounted for could reinfect the network and therefore needed to be identified in a hurry.
To start the process, Microsoft Information Scan was used to remove the worm from Exchange. Next, Trend Micro Inc.'s ScanMail antivirus software was deployed to stop the worm from breaching the perimeter. Wood says he used Tivoli products to scan his enterprise's 100 servers and 1,750 desktops in 128 locations for infected systems and to clean those in need with a distributed package of scripts created by the team. Using NT logging, the scripts made sure an unknown machine could not log on to the network until it verified its end point location.
Following that, new Sophos antivirus software was installed on remote machines from a centralized system. New rules were added to pay close attention to potential worm-carrying agents such as Visual Basic Script files, ZIP files, and .exe files.
In less than a week, Wood says his system was 90 percent to 99 percent up and running and clean, without a single project time line or federal appropriation being jeopardized.
"I don't think any of us really expected this kind of virus attack to be as smartly written, as virulent," Wood says. "I'll admit, personally, I was caught by surprise."
Wood says security technology is playing a bigger role in his department these days. He is determined to have checks and tools in place so he won't have to dodge a similar bullet.
"A lot of us are getting a little bit long in the tooth and gray in the beard.
We need to pass something to our successors for them to understand how we dealt with this process."