Every morning, Mark Bialik diligently scans 30 to 40 vendor and security Web sites looking for the latest patches for Infinity HealthCare's many Microsoft Office, Web server and Windows NT-based applications.
On a good day, it takes two hours to conclude that no new patches have been issued. On a not-so-good day, Bialik spends up to four hours figuring out if he's running the affected software. In the worst case, such as when the Code Red worm hit last July, Bialik's day is consumed by installing patches for 20 servers and hundreds of PCs. "I remember a time when I spent three to four days straight doing nothing but patches," says Bialik, a network and security manager at the health care provider.
These days, new security software makes it easier to distribute and test patches. But finding a fast and reliable way to identify new patches and prioritize installation remains elusive and costly.
Companies spend more than US$2 billion annually on patch research and deployment, according to Aberdeen Group Inc. Meanwhile, the pressure to find and install every patch is increasing as companies heighten security and focus on intrusion detection and managed vulnerability scanning.
It has to be done, so how can systems administrators and security managers make patch management more manageable? Security software vendors, end users and analysts offer the following three tips:
1. Develop a "Patch Network" Security software products can help streamline the process of finding patches by offering links to vendor sites. But vendors have come under scrutiny for not releasing patches fast enough for their users. Problems and patches can be more quickly identified by establishing a network of peers in multiple organizations, such as former colleagues or people at like-minded institutions, says Eric Hemmendinger, an Aberdeen Group analyst. "They may be your best resource," he adds.
Security portals such as Sans.org and Incidents.org also provide a front line for identifying patches and fixes. "Find good, reliable places that gather the data for you, and make a habit of reading them daily," Bialik says.
2. Buy Time by Prioritizing Before rushing to install every patch that comes along, prioritize installations according to their impact on the organization. A vulnerability in an e-commerce application should take priority over one in a platform that's fairly well hidden from the Internet, for instance.
If a high-priority vulnerability is identified, security managers are finding that multilayered security software, which is located at the firewall as well as the lowest level of the network stack, can temporarily plug the hole until a permanent patch is installed.
"Customers recognized this benefit before we did" acknowledges Jon Greene, senior vice president at Network-1 Security Solutions Inc. which sells a line of software security products. "If the intrusion can be detected, we can identify it and stop it. That buys them time to assess the appropriate patches that need to be deployed."
No matter how critical the patch may be, don't rely on fixes offered at hackers' Web sites; they can't be trusted. Bialik offers this advice instead: "If you can get by without running that particular application for the time being until the fix is out, turn it off!"
3. Evaluate Before You Patch To save yourself time and legwork, invest in security software that keeps a log of patches installed on each PC and server. The software can also check to make sure patches are working and will rank the vulnerability of each application.
Klipsch Audio Technologies uses St. Bernard Software Inc.'s Update Expert to identify servers and PCs that need patches, scheduling upgrades after business hours. "Something that would've taken six people four hours to do, we can set up in 10 minutes and not have to worry about it," says Mike Fulton, a network manager at the Indianapolis-based audio systems manufacturer.
Another tip: Test the patch first in a development environment to make sure it won't create new problems with the rest of the system. Companies that don't have the luxury of a complete test environment can develop a scaled-down version with at least a copy of the operating system running the applications in production.
And finally, beware of the pitfalls of patch-management software. Users report confusion over which security patch service packs work with different software versions. They also tell of technical support staffers who refused to help with a patch because their companies weren't running the latest version of the vendor's software. Other users say some scanning software can give false positives on uninfected machines.
Do's and Don'ts
DO establish a network of peers outside your organization to help identify vulnerabilities and find patches.
DO prioritize installations according to their impact on your organization.
DO invest in security software that keeps a log of patches installed on each PC and server.
DON'T rely on quick fixes offered at hackers' sites.
DON'T install patches without first testing them in a development environment.