When we tested four next-gen firewalls strictly on performance, we found that the products could forward packets at impressive rates, but throughput dropped when advanced security features were turned on. We now dive deep into application identification and control - the defining features of next-gen firewalls - to find out what works and what doesn't.
We discovered that although the four products tested show promise, there's still work to be done. Check Point, SonicWALL and Fortinet were clustered at the top of our scorecard, but still have areas we hope to see improved. Barracuda didn't score as well, but is in the middle of a significant product upgrade.
The defining characteristic of a next-generation firewall is the ability to identify and control traffic at the application layer, so we designed a suite of 40 tests in nine categories to see how well the firewalls lived up to their billing.
No one came close to a perfect score, with SonicWall SonicOS identifying and blocking 26 of our 40 test applications, followed closely by Check Point Security Gateway with 24, Fortinet FortiGate with 21 and Barracuda NG Firewall with 18.
(Editor's Note: In the first part of this test, vendors submitted their biggest, fastest boxes to David Newman's lab in California for performance testing. We allowed vendors to send a smaller, lighter device within the same product family to Joel Snyder's Arizona lab for features testing. In every case except SonicWall's, the actual product name was the same for both tests, just a different model number. In SonicWall's case, we tested the SuperMassive 10800 for performance and the NSA E8500 for features, so to avoid any confusion we're referring to the product here as SonicOS, the operating system both models share.)
In our testing, some apps caused more problems than others. For example, in our quest for recent episodes of "The Big Bang Theory" (porn for geeks), Check Point and SonicWall blocked our BitTorrent client from reaching out and touching Sheldon, while Barracuda and Fortinet didn't.
On the other hand, Check Point couldn't block Skype and none of the products blocked Google's Gmail, which slipped through when we hit the "click here for basic HTML if your browser is not showing you your email" button.
SonicWall has so many sub-divisions of every application, none of which were documented or made any sense to us, that we gave it a failing score when we tried to allow end users to see Facebook, but not post to it — one of vendor marketing's favorite examples of why a next-generation firewall is a good idea. It was possible to block Facebook completely, but you can do that with a URL filter — you don't need a next-generation firewall. SonicWall would have had a higher score if its application identification GUI wasn't so poorly designed.
The Check Point Security Gateway has a fantastic management interface for application identification and control that is much easier to use than the other products we tested. However, the engine underlying that interface doesn't work as well as SonicWall. For example, we could easily create policies that blocked particular parts of Facebook or LinkedIn, but those policies didn't actually work. Only when we blocked all of LinkedIn, for example, did the firewall behave properly.
Fortinet's FortiGate fit somewhere between SonicWall and Check Point on the management interface front. Not as elegant as Check Point, but much more usable than SonicWall, FortiGate was easy to learn and use.
But FortiGate stumbled most when encrypted traffic was involved. For example, a rule to block the popular webmail application Squirrelmail worked great when Squirrelmail was run over standard Port 80, but if we encrypted the same traffic on standard HTTPS Port 443, FortiGate wouldn't block it — even though we could see that the FortiGate was decrypting and re-encrypting the traffic as expected. The same was true of Facebook — unencrypted Facebook was blocked or allowed per policy, but if we simply used HTTPS for Facebook, the policy didn't work properly.
We had a difficult time making Barracuda's next-gen firewall block applications without some help from technical support, largely because of the poor design of the management GUI.
For example, because application identification occurs in the HTTP and HTTPS proxies, which are separate tools, you have to duplicate policy, wasting time and adding the opportunity for errors and inconsistencies. Barracuda told us that this, and other problems we had in the GUI, would be fixed in release 5.4, so we advise waiting until that version is available before even starting to test next-generation features.
Even if you do remember to change the policy in both proxies in the Barracuda NG Firewall, you also have to be careful when defining applications to be blocked. Although you get to pick which application you want to block in the first screen that pops up, you have to scroll down for three full screens before you can enter the list of networks this rule applies to.
Apparently, if you leave that blank, it doesn't apply to any users or networks, nor is there any pop-up dialog box saying "you've created a new rule that doesn't actually do anything.''
Overall, Barracuda turned in the lowest application identification score because it didn't have the ability to match as many applications as we were testing for. For example, the NG Firewall didn't have signatures for generic webmail applications or tools such as Lotus Notes, Outlook Web Access or SharePoint.
Some of the application categories the NG Firewall did have didn't make a lot of sense to us. For example, to block YouTube, you have to block "social networking," which does work — but it blocks more than just YouTube.
And when a category was successfully identified, the NG Firewall didn't always successfully block it. For example, Microsoft and Apple software updates showed up in the logs when we added a rule, but the NG Firewall wasn't able to successfully block them.
Additional features
The demand for next-generation firewalls may be focused on application identification, but we believe that there are other ways to "widen the tuple" to help network managers classify and control traffic. For example, we found that all four of the products we tested let us add user or group information to policies.
We were interested in other ideas so we went looking for reputation-based policies, rate-based policies, and geography-based policies. For example, a network manager might want to block some applications, such as outbound FTP, to or from particular geographic areas (if you're willing to trust that GeoIP databases have a low error rate, which isn't necessarily true).
Fortinet's FortiGate lets you write rules that refer to geography rather than just IP addresses. But more often, these features were not integrated into the firewall rule base. Check Point and SonicWall, for example, both allow the network manager to control traffic based on both IP reputation and geography, but did not fully integrate this feature into the firewall rule base; FortiGate has a slick rate-based policy feature designed to avoid denial-of-service attacks, but didn't integrate it into their firewall rule base.
It's a little early in the world of next-generation firewalls to say what else should go into firewall rule bases beyond application and user identification, but our testing showed that engineers are thinking about different options in this area.
Another area still out for discussion is exactly how application (and other next-generation) controls are integrated into the next-generation firewall. One school of thought suggests that they should be folded directly into the firewall rule base, creating a single unified policy that can refer to IP addresses and ports, users, and applications all at once. The other approach seems to be pulling application controls out into a separate rule base.
In testing four products, we found four approaches to this question. All four left user-based (and user group-based) controls in the main firewall rule base. From there, though, we found lots of variation. The Fortinet approach integrates everything into a single rule base, which we found the easiest to manage and the most intuitive from a basic security point of view. This approach is potentially the most powerful because it allows traffic to flow only when all attributes match up, and it allows you to interleave rules with and without application controls.
Check Point and SonicWall broke the application rules out from normal firewall rules, meaning that traffic must first pass through the firewall rules and be allowed before any application controls come into play. In the Check Point model, application and URL filtering rules are integrated into a single rule base, while SonicWall has a standalone application firewall module.
Barracuda's NG Firewall puts separate application rule sets in their HTTP and HTTPS proxy software. We found this problematic, not only because you have to define duplicate policies, but also because the way that policy definitions are created makes it impossible to mix "pass" and "block,'' severely limiting the flexibility of the engine. Barracuda told us they had this slated for a fix in Version 5.4.
Check Point and SonicWall engineers both defended their choices by pointing out a problem in application firewalls: you can't decide what application is being run without allowing some traffic through the firewall, including, perhaps, traffic that the network manager might want to have blocked.
An example might be helpful. Suppose you have a corporate policy that says "SMTP outbound is allowed only on Port 25." Taken in isolation, this means that you'd have to write two rules, one to allow SMTP on Port 25 and the other to block SMTP on all other ports. Then, you could have additional allow rules, such as allowing HTTP or IM outbound traffic on multiple ports. The result of this policy would be that the firewall would have to allow all other traffic outbound to connect and transfer data long enough to decide whether or not it was SMTP traffic. Our vendor engineers were concerned that this could easily result in unintended consequences and insecure configurations — a valid concern.
This is one area where next-generation firewall vendors are still finding their way. We think that Fortinet is on the right track here, but since this is an open area of discussion, we did not include it in our scorecard.
Taking action
The last area we looked at was the action options. In our testing, we simply asked the firewalls to block traffic. But in the case of web-based applications, the network manager might want to intercept the request and display a page to the end user indicating that security policy prohibited the transaction.
The Check Point Security Gateway, which integrated URL filtering with application identification, was the only product that included this feature. The Security Gateway actually goes further than that, allowing the next-generation application identification rule to have an action that displays the "page blocked" message while allowing the user to click on through after acknowledging a warning.
We found other options as well. For example, SonicWall and Fortinet let an application rule apply some QoS settings, such as limiting traffic (in the case of bandwidth wasters, for example) or guaranteeing traffic (in the case of VoIP or video conferencing). Both also allow an action of "log packets" to save a transaction for later analysis.
When it comes to actually identifying and blocking applications, what we would prefer is a hypothetical product mixing two of the devices we tested: the SonicWall SonicOS engine configured by the Check Point Security Gateway management system. In the absence of such a mythical beast, SonicWall did the best job of identifying and controlling applications, but we found room for improvement in everything we tested.
Of course, application awareness is the icing on the cake for next-generation firewalls, which also have to handle all of the other basic tasks associated with a state-of-the-art firewall. In this next section, we delve into those other functions, which we've divided into network visibility, SSL decryption, IPS, UTM and basic firewall blocking and tackling.
Snyder, a Network World Test Alliance partner, is a senior partner at Opus One in Tucson, Ariz. He can be reached at Joel.Snyder@opus1.com.
Read more about wide area network in Network World's Wide Area Network section.