When it was time to write this column, the only thing on my mind was the OpenSSL Heartbleed vulnerability. If you have anything to do with infosec, it was probably dominating your days as well.
Trouble Ticket
At issue: Heartbleed is a major vulnerability that needs immediate attention.
Action plan: Set priorities and systematically check everything.
If by some chance you haven't heard, a vulnerability was published in early April explaining a way to take advantage of a coding error in the way OpenSSL keeps a session opened. The vulnerability allows for the disclosure of up to 64KB of memory, which may contain data such as user IDs, passwords, secret-key material and other sensitive data that may reside in memory.
Sometimes when a high-profile security vulnerability is released, I try to gauge the hype against reality. For Heartbleed, I got my hands on exploit code and ran it against some servers that were running a vulnerable version of OpenSSL.
The exploit code was quite easy to compile and run. You simply run the exploit against an IP address and port. If there is any data in memory, the results are displayed. The results were amazing. For one of our internal high-traffic Web servers, I ran the exploit several times before I was able to capture a username, but that was still all the convincing I needed to take action. My priorities at that point were to check on the security of my company's products and services, our internal infrastructure and the many systems we use that are provided by vendors, especially software-as-a-service (SaaS) offerings.
I was relieved to find that our customer service organization was already identifying all of our products that use the vulnerable version of SSL. That team is remediating all products and services, and we have placed an announcement on our support portal so our customers know the status regarding this vulnerability. I will follow up with independent assessment.
Next on the list was our internal infrastructure. This includes routers, switches, firewalls and other network and security devices, as well as internal applications and servers. Our approach here was two-pronged: First, ask each vendor about any known vulnerable products or services. Second, conduct scans of our IP address space.
The scanning took quite a while. We wanted to be thorough, so we used Nessus to scan all the ports of every IP address -- a total of 65,535 ports.
Finally, we contacted all of our vendors, notably the providers of the 150 or so SaaS applications we use. To speed this process, I gave each in-house application owner an email template for the queries. For immediate peace of mind, I also had my team run a tool, provided by one of the certificate vendors, against all SSL-enabled Web services. The tool indicated whether a site was potentially vulnerable by retrieving certificate information from the server, which included the version of SSL.
What we have found out so far has been interesting, and at times alarming. Many of our network and security vendors have issued statements regarding vulnerable infrastructure we use, and some have already issued patches. Other vendors are still assessing the situation. Scans of our internal infrastructure yielded quite a number of servers that are vulnerable. Interestingly, we discovered that more than 300 resources that run Windows Server are vulnerable. That had us scratching our heads until we realized that it was the Integrated Lights Out board, used for remote server management, that was vulnerable. We are working with the vendor to obtain a patch.
We also discovered some 40 vulnerable servers on our PC network. This was traced to users who were running vulnerable virtual machines on their PCs.
The work continues, and I will provide status reports to our executive staff on a weekly basis until all issues have been remediated.
This week's journal is written by a real security manager, "Mathias Thurman," whose name and employer have been disguised for obvious reasons. Contact him at mathias_thurman@yahoo.com.