The controversial 'Deep Throat Fight Club' test of porn filters held at this week's RSA security show has declared a winner. According to organizers Untangle, the best performer was Fortinet.
Fortinet detected 97.7 percent on the main blocking porn test, only a whisker ahead of rivals Watchguard (97.3 percent), Websense (97.0 percent), SonicWall (96.1 percent), and Barracuda (94.0 percent).
On the 'porn categorization' test, which measured performance against a human making judgements, Fortinet again came out top, again the margins were small, the percentages almost identical, and the order of success exactly the same.
In other words, it depends how you look at the results. Is the fact that all products blocked at least 94 percent of porn requests in the automatic blocking tests good, or the fact that they missed up to six percent bad?
Already, Watchguard - which came a very close second in the tests - has questioned the methodology, pointing out that URL filtering can be circumvented in a number of ways. The methods are startlingly diverse. Porn hunters can use everything from short URLs, website proxying, and even log into https-based sites such as penthouse.com and not be noticed by standard URL filtering, Watchguard has claimed.
Untangle's tests were based on googling 50 porn search terms, choosing ones that were 100 links deep against which to measure the filters. The products were then graded against a human making the same judgements. The full list of sites has been included in a spreadsheet that can be downloaded from the company's website (30MB download).
To be fair to Untangle, the open source company probably intended the tests to present a baseline level of URL filtering effectiveness. Any technology can be bypassed if the user has the knowledge to do so.
"If you are expecting to block all porn, all these products will not be effective, however if you're looking to control and monitor porn usage as much as possible these products make great candidates," wrote Untangle's Dirk Morris, in his post-test blog.
Watchguard's objections are technically valid, but they tend to undermine the point of using porn filtering at all. That said, the company says its firewalls would also work against the bypassing technologies, something not measured in this 'bake-off'. Perhaps what is now needed is a new Fight Club test to look into this aspect of filtering.
Untangle has at least answered the question as to why it had not included its own URL filtering system in this week's Deep Throat Fight Club test.
"We felt including Untangle itself in the test would not be fair, as we know the test set and methodology ahead of time. We also apologize to the vendors (netsweeper, besafe) who requested to be in the test but time didn't allow," said Morris.