The German firm AV-TEST today stood by the results of its search engine investigation that claimed Microsoft's Bing shows five times the number of malware-hosting websites than Google in its results.
On Friday, Microsoft called AV-TEST's results flawed.
"AV-TEST's study doesn't represent the true experience or risk to customers," alleged David Felstead, senior development lead for Bing, in a Friday blog.
The website that Felstead cited as an example of how Bing warns of dangerous destinations was, its owners claimed, free of malware and had never been compromised in its 14 years on the Internet.
In his blog post, Felstead reacted to a report issued April 6 by AV-TEST of Magdeburg, Germany. The report said Bing indexed and returned in its search results nearly five times as many malware-infected links as Google.
Over an 18-month stretch, AV-TEST evaluated more than 40 million websites to determine the extent of a long-held maxim by security professionals: That even with extensive efforts to scrub search results of dangerous links, engines such as Google and Bing cannot stop cyber criminals from exploiting search tools -- and users' reliance on them -- by either compromising legitimate sites or artificially promoting malformed websites to host attack code.
"Google achieved the best results in the study, followed by Bing," said AV-TEST in its conclusions ( download PDF). "Attention must, however, be drawn to the fact that Bing delivered five times as many websites containing malware as Google during the study."
According to AV-TEST, of the 10.9 million tested with Bing, 1,285 were found to host malware, for a infection rate of 0.012%, or 12 sites out of every 100,000.
Of the 10.9 million websites tested with Google, 272 contained attack code, an infection rate of 0.0025%, or 2.5 sites out of every 100,000.
Those infection rates may be minuscule, but AV-TEST argued that in practicality, the number of malware-hosting sites encountered by users was significant simply because of the volume of queries run each day on the major engines.
"It is important to remember that Google alone deals with a phenomenal total of 2 to 3 billion search requests worldwide every day," AV-TEST said. "If this total is factored into the calculations, the total number of websites containing malware found by the search engine is enough to make your head spin!"
Microsoft took nearly two weeks to respond to AV-TEST's claims, but when it did, it pulled few punches. "The conclusions many have drawn from the study are wrong," Felstead said flatly.
Felstead based his argument on the warning that appears when links suspected of harboring malware appear within Bing's results and those links are clicked by the user.
"By using the API instead of the user interface, AV-TEST bypassed our warning system designed to keep customers from being harmed by malware," said Felstead. "Bing actually does prevent customers from clicking on malware infected sites."
Felstead said that users see the warning only once in every 10,000 searches, or 0.01% of the time, a number close to AV-TEST's 0.012%. "In any case, the overall scale of the problem is very small," Felstead asserted.
AV-TEST confirmed today that it relied on a Bing API (application programming interface) to collect search results from Microsoft's engine.
"No links were clicked/followed through the search engine," Andreas Marx, CEO of AV-TEST, said in a Monday email reply to questions. "We simply grabbed the URLs and downloaded them on our own systems for further analysis. We didn't want to test the warnings from the search engine but simply how many potentially malicious websites are returned by the search engine."
Microsoft cited vacationhotline.net as an example of how Bing warns users. But the site's owners denied the infection allegation. Computerworld confirmed that Bing shows this warning. (Image: Microsoft.)
Marx acknowledged that some search engines ward off users from suspicious links with warnings, but of those, not all are as clear as Bing's, and could be easily ignored or dismissed by users.
He also cited Felstead's claim that "Our data shows that these warnings block 94% of clicks to malicious sites" to defend AV-TEST's approach.
"Microsoft argues that their warning is 94% effective, so 'only' 6% of the people will click on the malicious link anyway," Marx countered. "Still, that's a lot of people."
Additional protective measures, including Bing's in-results warnings, those displayed by browsers -- all the major Web browsers have mechanisms for warning users of potential danger when they click on some links -- and others generated by security and antivirus software, were "out of scope for this study," Marx said.
Marx stood by the study, and said it would not be revised to take Microsoft's complaints into account. "The report was NOT designed to be a 'safety comparison' for search engines," he said.
However, he confirmed that AV-TEST was considering revamping its methodology for future tests. "In the next report, we might be able to report that search engine A warns [of] 30% of the malicious links, or if it's just 1%, or more like 70-80%, plus how many false positives we've seen," Marx said.
As to why Bing indexes suspicious links and shows them in its results, Felstead contended that "most are legitimate sites that normally don't host malware but have been hacked."
He also called on other arguments to defend Bing's approach, including one related to competition with Google and other engines. "We warn our customers rather than suppressing the result [because] if a user searches for 'vacation hotline' and doesn't get the site they're looking for, they perceive Bing to be an incomplete index of the Web which impacts their confidence of the engine," Feldstead said.
Feldstead cited the website vacationhotline.net to show Bing's warnings.
But Thomas Stelter, CFO and a co-owner of Foremost Travel & Tours of Chicago -- which manages vacationhotline.net along with scores of other travel sites -- categorically denied that the domain was infected.
During a telephone interview, Stelter used Norton Safe Web -- a component of Symantec's security software -- to examine the link to vacationhotline.net and reported that the Norton tool said it was safe to click. Other tools at Foremost's disposal also showed no evidence of malware.
Stelter was unaware that Bing had classified his website as hosting malware until the call from Computerworld.
"This creates a significant concern on our side when proper reporting and testing has not been done," Stelter said after collecting himself. "Norton is telling me it's safe, Bing tells me it's not. This inaccurate analysis causes further confusion within the consumer marketplace about where I should go or not go."
Stelter said that his firm manages more than 100 travel-related websites, has a significant presence on the Web -- vacationhotline.net was first registered in 1999, virtually in the Internet's Dark Age -- and has never before been accused of harboring malware.
Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer, on Google+ or subscribe to Gregg's RSS feed. His email address is gkeizer@computerworld.com.
See more by Gregg Keizer on Computerworld.com.
Read more about malware and vulnerabilities in Computerworld's Malware and Vulnerabilities Topic Center.