When you install antivirus protection to clean up a malware-infested PC, two things should happen. Firstly, the antivirus should detect the active malware; and secondly, it should thoroughly remove the infestation and rectify its effects. The latest test from AV-Comparatives specifically evaluates how well an antivirus product manages the clean-up process. Overall, this year’s results look better than the introductory run of this test in 2011.
This time around, the list of products tested is rather different to last year. AV-Comparatives only included products whose vendors subscribe for testing at the Premium level. K7, McAfee, Microsoft, Qihoo, Sophos, Trend Micro, and Webroot were tested last year, but as they subscribe at the Basic level they didn't make the cut this time. BullGuard, Fortinet, and GFI Vipre are new additions, not tested last year. This year's test looked at 13 products in all.
Symantec's researchers have a beef with testing methods in one particular AV-Comparatives test. AV-Comparatives won't let vendors pick and choose which tests they'll participate in, so this year Symantec isn't participating at all.
Because this is very specifically a test of malware removal, the researchers only chose malware samples known to be detected by all of the tested products. They further ensured that each sample is found in the wild, and that it doesn't perform irreversible destructive actions. Finally they narrowed down the possibilities to 14 samples that demonstrate common malware behaviours.
The researchers thoroughly analysed each sample, recording every change made to the infested system. For each product, they installed the sample on a clean test system, rebooted, and verified that the sample was active. Next they installed and updated the antivirus and ran a full scan. After rebooting, they checked the test system manually to see how well the antivirus product cleaned up.
When you install antivirus protection to clean up a threat, you want to just run a scan and be done with it. A product that completed the clean-up process under Windows there and then earned an A for convenience. If installation or clean-up required rebooting in Safe Mode or running a separate utility, that product got a B. Most of the tested products offer a Rescue Disk for tough cases. Resorting to the Rescue Disk isn't very convenient, so products that required it get a C. Finally, if the clean-up utterly failed or couldn't be completed without intervention by tech support, that means a D is awarded.
After each product did its best to clean up each sample, the researchers dug in and manually checked to see how good a job it did. To earn an A for clean-up, the product had to remove everything except negligible, non-executable traces. If the product did disable the sample but left behind significant traces like executable files or changes to the MBR, that earned a B. A product that removed the malware but left annoying or dangerous problems rated a C. Examples of such problems include a disabled Task Manager, disabled Registry Editor, and compromised HOSTS file. If removal failed or the system became unusable, the product got a D.
To get the final score for a given combination of antivirus and malware samples, the researchers used a simple scoring system in which A for removal and A for convenience is worth 100, A for removal and B for convenience is 90, and so on down to 0 points for D and D. A product's final score is simply the average across all fourteen samples.
Bitdefender Antivirus Plus 2013 and Kaspersky Anti-Virus (2013) managed an A average in both removal and convenience. That comes as no surprise to me; both products also scored very well in my own malware clean-up testing. Panda Cloud Antivirus Free Edition 2.0 came close with an A- in both categories. All three of these products earned the top rating, ADVANCED+.
Even the lowest scores weren't dreadful. AVG Anti-Virus Free 2013 earned a B average for both thorough removal and convenience, while Avast! Free Antivirus 7 averaged a B for convenience and a B- for thorough removal. These got a STANDARD rating, the lowest passing rating. All of the others were rated ADVANCED.
Overall, the products in this test did better than in last year's test. Last year 10 of the 18 products tested only reached the STANDARD level. ESET NOD32 Antivirus 5, F-Secure Anti-Virus 2013, and G Data AntiVirus 2012 all went from STANDARD last year to ADVANCED this year.
The full report goes into significant detail describing the malware samples used, and lists the precise score for each product with each sample.
It also lists a number of ways that antivirus vendors could improve their scores, and thereby improve their product's ability to help customers rid themselves of malware. For example, vendors should provide an offline installer in case malware interferes with a connection to the vendor's website. Antivirus installers should check for and deal with active malware before attempting installation. The antivirus tool itself should point to standalone tools if installation or clean-up fails. Vendors, take note; these are really good ideas!