Microsoft Not Happy With Test Results OneCare, Questions Secunia’s Testing Method

In the latest test by Secunia, Windows Live OneCare ended up fifth, detecting only 1,67% of the exploits tested on. The test cases were a mix of three different kinds of exploits: Proof of Concept (PoC), GameOver PoC and Exploit. The test cases were separated into two groups, malicious files (.gif, .mov etc.) and malicious web pages (ActiveX and other browser vulnerabilities).

secuniaresults_thumb_39d82346 News

Not happy with the results Microsoft labeled the results as misleading and confusing to average consumers.

“At first glance, it would seem that this test would be very helpful to consumers to determine the best security suite in the specific scenario – although none of the suites did particularly well in this specific situation. However, Secunia’s test focused only on the on-demand scanner functionality and did not take into account any of the other built-in security protections in Windows Live OneCare or other suites included in the test.”

“The on-demand scanner functionality in Windows Live OneCare is a useful tool, but it is only one piece of the overall solution. To be fully effective, it must work in conjunction with other functionality in the suite including, but not limited to, the real-time anti-malware detection engine, the firewall, and automatic update engines like Windows Update and Microsoft Update…”

Did they really only consider the on-demand scanner? How did they conduct these tests?

From the test results:

The testing process consisted of the following steps:
1. The malicious files were first tested by unpacking a ZIP archive containing the files in order to test
the efficiency of real-time access scanning.
2. Then the folder was scanned manually to ensure that all files were scanned, regardless of any policy
limitations on the real-time scanning.
3. Malicious web pages were tested using Internet Explorer to visit the individual pages one by one.
This was done using regular http connections to ensure that none of the products would be foiled by
encrypted https traffic (even though a good product should not be affected by this).

I see real-time protection and on-demand there…so ehh slight misreading/miscommunication there Microsoft? So what else is wrong with the way they test?

“Moreover, Secunia’s test explicitly focused on machines that were unpatched with some of the latest updates, both to the Windows operating system and to applications on the machines.”

Let’s see what got tested on what exactly:

All tests were carried out on Windows XP SP2 missing certain patches and with a number of vulnerable
programs. ZoneAlarm was tested on an SP3 machine due to compatibility issues.

Yep, the test were indeed performed on non updated machines, but is that necessarily wrong? We always stress people to update their machines and software because of possible vulnerabilities, but how many really do? In the real world how many computers are fully up-to-date? According to Secunia almost one-third of all installed software lack one or more security related updates. Should these Security vendors dismiss exploits just because there already are patches to fix them? Sometimes doubles are wrong but in this case I’d rather be double protected than not at all, patches and AV/Security definitions do not bite each other!

Anything else possibly wrong?

Out of the 300 test cases, 126 are considered particularly important. These 126 test cases affect very
popular products and have either been discovered as zero-day threats, public exploits exist, or Secunia has
developed working exploits.

Note: Secunia does not usually develop working exploits as the Secunia Binary Analysis service is defensive
in nature; thus working exploits are not necessary for developing and testing signatures. Generally speaking,
Secunia focuses on developing PoCs for the analysed vulnerabilities, since these are better suited for
signature development.

Working exploits developed by Secunia? That I do find somewhat questionable, how is an AV program supposed to protect one against exploits not yet known (or modified) to the program vendors? No program can do that, honestly! Even if you keep everything up-to-date, new viruses and exploits are developed every day, so you can never be 100% safe!

Windows Live OneCare is tested by numerous organizations around the world and has been certified for anti-virus protection by two of the industry’s leading independent certification authorities: International Computer Security Association Labs (ICSA) and West Coast Labs. In addition, Windows Live OneCare, along with Microsoft Forefront Client Security, has continually successfully received VB100 awards since June 2007.

The WildList Organization compiles virus reports from anti-virus experts around the world on a monthly basis to track against viruses currently spreading throughout the user population. Microsoft prioritizes these wild list tests as some of the most important measures of OneCare’s performance against real world threats, and not just simulated malware that may or may not cause real harm. Furthermore, to achieve the VB100 award, Windows Live OneCare was also deemed to be able to accurately detect the complete set of the “in the wild” malware without triggering any false positives.

That’s all good, real “in the wild” malware, not created by Secunia specifically for testing purposes. The test is what it is, an Internet Security test and none of the suites did well. It did test real-time protection, which is the most important bit of Security software.

Still Secunia’s conclusion that the major security vendors do not focus on vulnerabilities stands. Let me repeat: should they dismiss them because patches already exist, or should they consider those consumers who may not have their computers updated besides the definitions of their AV software?

Windows Live OneCare
Windows Live OneCare Spaces – Understanding Test Results for Security Solutions
Secunia – Internet Security Test October 2008 Test Results (pdf)
Softpedia – Windows Live OneCare Detects Just 1,67% of Exploits