Thought you'd hit every item on the year 2000 checklist?
Add one more: Check the Y2K repair work, including the installation of patches, that you think has been done by outside vendors or systems integrators.
That's the lesson The Boston Children's Hospital learned on two systems maintained by outside vendors. While the hospital's mission-critical systems tested fine for Y2K, bugs turned up on secondary systems.
One case involved a software vendor and reseller the hospital paid to upgrade the Hewlett-Packard Unix operating system that runs a radiology record-keeping system. That outside vendor provides the record-keeping software, hardware and operating system under a turnkey arrangement and does upgrades through a maintenance contract, according to Stuart Cohen, manager of systems for the hospital.
The vendor, which Cohen declined to name, performed an upgrade to the HP-UX operating system in August in preparation for an upgrade of the record-keeping application later that year, and all appeared to be going well.
Hospital IT staffers ran the record-keeping application and systems software through the hospital's Y2K acceptance test, and the system passed, Cohen said.
But last week, when IT staffers tested the system using an HP Y2K tool that they found through HP technical bulletins, they learned that several Y2K patches hadn't been installed when the outside vendor upgraded HP-UX from version 10.01 to 10.20. "The testing that we have done on systems has not revealed any problems before, so the nature of the non-compliance is likely to be very minimal. It's probably not a feature of the operating system that is used by the application," Cohen said.
But HP still recommends that the hospital get the Y2K patches, and what's aggravating, Cohen said, is that those patches had been available since April, long before the outside vendor did the upgrade. Now, the vendor told him it will cost $US4,000 in labour costs to install the patches.
Cohen doesn't want his own staff to install the patches because "we wanted the vendor to own the whole issue. At this very late date, we didn't want to risk that the patch would create some profound change to the operating system."
Cohen is scheduling the work for 2am on December 30. He said he will have to wait until later to deal with the contracts matters.
"If [the patch] was available, they should have done it," Cohen said. "Now, after the fact, to be charging me doesn't seem reasonable."
Cohen said the hospital is investigating why the vendor didn't install the patches and searching for the contract to find out what was written about Y2K compliance.
"This has been an ongoing issue since Day One. Many [vendors] are saying Y2K patches are outside your maintenance agreement. I think it's pretty clear this is a maintenance issue," Cohen said. "I didn't buy software with a December 31, 1999, expiration date."
In another case, Cohen says, a systems integrator failed to install one or two Y2K patches for Windows NT servers in the hospital's cardiology department, when it was migrating from Novell to Microsoft server software. Cohen said the systems integrator's contract called for it to install Y2K patches.
"We paid them on a time-and-materials basis to do the work, and I just know they're going to turn around and tell me that it's going to be time and materials to finish the work," Cohen said. As with the software vendor, Cohen declined to identify the systems integrator, so Computerworld was unable to contact it for comment.
Cohen said he has learned that "it's a good idea to check" even when you think systems integrators or outside vendors have done the Y2K work.
Among the other precautions the hospital plans to take is shutting off incoming network traffic from the Internet starting at 10pm on December 31. Internet e-mail will be held back at the firewall and queued for delivery when IT staffers deem it's safe to open the gateways again.
"E-mail is the primary source of viruses. So cutting off e-mail before midnight seems prudent," Cohen said, adding that the staff will be carefully monitoring news reports for potential problems.
One precautionary measure that the hospital hasn't seen the need to take, however, is pre-declaring a disaster so it can reserve space with the disaster recovery companies that take over systems in the event of a radical problem.
"All our efforts indicated we were making good progress," Cohen said, noting that outside consultants and auditors have been reviewing the hospital's Y2K effort for some time.
"We're preparing as if we're getting a Class 4 hurricane, checking equipment, supplies, generators, communications, backup communications capabilities, phones and pagers," Cohen said. "Our normal disaster recovery procedures are in place."