Wider implications of the Red Hat breach

Reports of data losses and system breaches are almost becoming passe but from time to time events happen that take on a life of their own and have effects far beyond what the initial breach would normally represent.

Late last week there was an announcement that key servers belonging to both the Fedora and Red Hat Linux distributions were compromised. With this breach they join the ranks of Ubuntu, Debian and Gentoo] as Linux distributions that have suffered severe server breaches. What is causing the most concern about Fedora's case is that one of the servers that had been breached was being used to provide authoritative signing of packages distributed under the Fedora banner. Had the attacker been able to capture the private key, or even the source phrase used to generate the key, then it would have been possible to generate their own packages that authenticated as official Fedora software. The Red Hat compromise resulted in custom OpenSSH packages being uploaded to the compromised server.

While Fedora have stated that they don't believe the key or phrase were compromised, many feel that it isn't good enough and are calling for Red Hat to be far more open in reporting exactly what happened. The different signing systems in use has helped mitigate the extent of the damage (otherwise Red Hat's compromise would have the same sort of risk as Fedora's) but there is concern about how readily the Red Hat system signed the modified OpenSSH packages.

It would be interesting to uncover the motivation for the attack. If handled carefully, the attacker could have subtly poisoned user-space applications that could have allowed the easy extraction of sensitive personal information for Identity theft/fraud purposes. Targeting key system components is more likely to have the attackers found out quicker, but it also means that the attackers would potentially have full system access to a large number of global systems without any extra effort. One day we will see a cross over point, where the value of quietly stealing the personal information outweighs the value of the system as part of a botnet and attackers will begin to focus on subtle user-level attacks to achieve this.

Page Break

Beyond the direct problems associated with resolving the breaches and uncovering the extent of any damage, the costs associated with re-verifying all source code and rebuilding packages make this particular set of system breaches more costly than most others. Questions should be asked about why the package signing machine was networked (and ultimately reachable from the Internet), is it a case where security has unintentionally been compromised for the sake of simpler administration and package distribution by remote authors?

Does being an open-source company make the recovery process any different than if it was a closed-source company? The public release of most of the source code to Half Life 2 (2003), Diebold Election Systems (2003), and partial source code to Cisco's IOS (2004), and Windows NT and 2000 (2004) didn't appear to have any major ramifications that could be linked to the breaches either in the period immediately subsequent to release or after.

If the attacker hadn't uploaded modified OpenSSH packages then the attack against Red Hat's systems might have gone unnoticed for a longer period of time.

There have been reports of attacks against SSH-enabled systems that come as a strange coincidence to this breach of Red Hat's systems. It may be that the attacks are related to the weakened Debian OpenSSH versions from a couple of months ago, but the timing suggests that the Red Hat breach is the more likely trigger for the event. With one of the goals of the infecting rootkit being to recover valid SSH keys from a compromised system it makes it harder to work out exactly how a breach has taken place when the attacker is able to use valid authentication credentials to gain access to a trusted system.

A question that all those responsible for systems and network security should be asking themselves now is just how much can they trust that their systems haven't been silently compromised, or compromised through trusted channels. Successful hacks are a matter of when, not if.

Even if you are running systems and application software that can be verified from the source, you still have to trust that at some level there is someone or some organisation that knows what they are doing and place your trust there. For a lot of users and administrators that run Fedora and Red Hat systems that trust is placed in the organisations responsible for the distributions and the servers associated with them. That trust, which in most cases is well placed, would have eventually resulted in widespread vulnerable systems if it hadn't been picked up on in this instance.

Ken Thompson's Reflections on Trusting Trust should be required reading for anyone who has to be involved in this sort of work.