Security company Imperva released a grim study last month suggesting that costly security suites may not be worth the price tag, and that all antivirus programs suffer from huge blind spots. Doom-and-gloom research like this always requires a hefty grain of salt, but after speaking with numerous industry experts, we’re starting to think that an entire shaker might be necessary here.
Imperva looked at a variety of security solutions from vendors such as Kaspersky, Avast, AVG, Microsoft, and McAfee, to name but a few. They pitted these sentinels against 82 randomly collected malware samples, examining how successful the security software was in detecting the rogue software.
From its work, Imperva asserts that anti-malware software is not fast or responsive enough to combat modern threats. Security software, Imperva claims, is "much better at detecting malware that spreads rapidly in massive quantities of identical samples, while variants that are of limited distribution (such as government sponsored attacks) usually leave a large window of opportunity."
They also found no correlation between the money users spend on virus protection and the security provided by the software, and suggested that both individual and enterprise customers look at freeware alternatives.
Independent labs push back
The study has attracted a lot of attention, but when speaking with security professionals, and some of the companies named in the study, we found many who believe the study to be deeply flawed.
Just about every lab or security company felt that Imperva's sample size of malware was too small to support the conclusions made by the study. AV-Test's Andreas Marx told us that his firm receives about a million samples of new, unique malware per week. Similarly, Peter Stelzhammer from AV-Comparatives told us that they receive 142,000 new malicious files each day.
For its part, Imperva wrote in the study that the firm intentionally used a small sample set, but insists that it is demonstrative of existing threats. "Our selection of malware was not biased but was randomly taken from the web reflecting a potential method for constructing an attack," Imperva stated.
NSS Labs research director Randy Abrams, however, had a sharply different interpretation of Imperva's methodology. "Searching for filenames is guaranteed to miss sophisticated attacks and most other malware too," Abrams said, commenting on the means Imperva used to locate malware for the study. "Focusing on Russian forums significantly biases the sample collection. It is obvious that no thought went into obtaining a real-world, representative sample set."
Problems of methodology
To carry out the study, Imperva used the online tool VirusTotal to perform tests, and this was cited as a critical weakness. "The problem with this test is that it ripped threats, in the form of executable files, and then scanned those using VirusTotal," said Simon Edwards of Dennis Labs. "VT is not a suitable system to use when evaluating anti-malware products largely because the scanners used in VT are not supported by additional technology such as web reputation systems."
Kaspersky Labs, whose product was used in the study, also questioned the testing methodology employed by Imperva in the experiment. "When scanning for potentially dangerous files, the VirusTotal service used by Imperva's specialists does not use the full versions of antivirus products, but merely relies on a standalone scanner," wrote Kaspersky Labs in a statement.
"This approach means that the majority of protection technologies available in modern antivirus software are simply ignored. This also affects proactive technologies designed to detect new, unknown threats."
Notably, a portion of VirusTotal's website discourages anyone from using its service for antivirus analysis. The company's 'About' section reads: "We are tired of repeating that the service was not designed as a tool to perform antivirus comparative analyses [...] Those who use VirusTotal to perform antivirus comparative analyses should know that they are making many implicit errors in their methodology."
Abrams also took a dim view on using VirusTotal to perform the study, saying that the tool can be used to skew results towards those desired by testers. He noted: "Competent, experienced testers know better than to use VirusTotal to assess protection abilities of anything other than a pure command line scanner.”
Imperva defended the use of VirusTotal in its study. "The essence of the report is not a comparison of antivirus products," the company said. “Rather, the purpose is to measure the efficacy of a single antivirus solution as well as combined antivirus solutions given a random set of malware samples."
While the experts we spoke with agreed that zero-day vulnerabilities and newly created malware are a problem, none supported Imperva's assertions about timing or low detection rates. "The lowest protection rates during a 'real-world' zero-day test are 64 to 69 per cent," Marx said. "On average, we saw a protection rate of 88 to 90 per cent for all tested products, this means 9 out of 10 attacks will be successfully blocked, only 1 will actually cause an infection."
Another key conclusion of the Imperva report was that anti-malware software is well understood by malware creators, who tweak their creations to subvert protection systems. "Attackers understand antivirus products in depth, become familiar with their weak points, identify antivirus product's strong points, and understand their methods for handling the high incidence of new virus propagation in the Internet," Imperva wrote in the study.
The study continues: "Variants that are of limited distribution (such as government sponsored attacks) usually leave a large window of opportunity."
Stuxnet isn't after you
"The malware guys are really tough, they're strong and intelligent," said Stelzhammer. "A targeted attack is always dangerous." But he and others stressed that targeted attacks where the malware is specifically tailored against anti-malware is as rare as it is dangerous.
The effort and information required to create a piece of malware to defeat every layer of protection is great. "Such a test requires a lot of time and skills, so they are not cheap," wrote Marx. "But that's the reason why they are called 'targeted.'"
On this point, Abrams quipped: "Frankly, I am really not concerned with Stuxnet getting into my computer and attacking a uranium enriching centrifuge at my home or employer's office."
Nearly everyone we spoke to agreed, at least in principle, that free anti-malware solutions could provide worthwhile protection for users. However, most disagreed that it was a viable option for enterprise customers. Stelzhammer points out that even if corporate users wanted to use free software, the licensing agreements sometimes prevent them from doing so.
"It's not all about detection," said Stelzhammer. "It's about administration, it's about rolling out to the clients, it’s about overview. You will not get this with a free product."
An informed user at home, continued Stelzhammer, could use layers of free software to provide protection comparable to paid software but at the cost of simplicity. "He can arrange a well-protected system with free software, but the biggest advantage of paid software is convenience."
However, Edwards of Dennis Labs disagreed with the favourable comparison with free software. "This is counter to all of our findings over many years of testing," said Edwards. "Almost without exception the best products are paid-for."
Since the publication of the study last month, Imperva has written a blog post defending the firm’s position. Imperva director of security strategy Rob Rachwald said: "Any critique focusing on our methodology is missing the reality we see today." He went on to say that most data breaches are the result of malware intrusion, which the company sees as proof that the current anti-malware model is simply not working.
While there may be some inherent truth to Imperva's conclusions, none of the experts we spoke with viewed the study positively. "Typically, I warn against vendor sponsored tests, but if this test had been performed by an independent organisation I would warn against the organisation itself," wrote Abrams of NSS Labs. "It is rare that I encounter such an incredibly unsophisticated methodology, improper sample collection criteria, and unsupported conclusions wrapped up in a single PDF."