New test methodology: Something went wrong

As we’ve written about before, we at Kaspersky Lab eagerly support a variety to benchmarking methodologies for security products. Ultimately, not only do benchmark tests show which solutions are the most effective, but they also serve to fuel constant product innovation and improvement. That is especially true of holistic research with testing scenarios reflecting real-life user behavior.

That’s why we were thrilled to learn that NSS Labs devised a new methodology for testing enterprise-class security products, which help to successfully detect, prevent, and continuously monitor threats at the endpoint level. In NSS Labs terminology, such solutions are called Advanced Endpoint Protection (AEP) solutions.

In mid-February, NSS Labs introduced the results of its pilot test using the new methodology. The products chosen for the assessment were 13 solutions classified as AEP, including the latest version of Kaspersky Endpoint Security for Business, which performed quite well. However, the report we got raised serious questions. The key reason: When we reproduced the tests, we got very different results. In fact, our test showed that Kaspersky Endpoint Security for Business underperformed twice, with the rest of the test scenarios yielding good results. What’s also unfortunate, we did not have enough time to discuss the performance with NSS Labs analysts (a common practice) before the report came out. After all, tests can fail, too. All in all, such a hastily prepared report could be explained by a rush to publish the results in time for the RSA conference.

We are now discussing the methodology with NSS Labs and together trying to figure out what could have gone wrong. Currently, we think that the new testing method requires some additional work. However, we are assured the problems will be solved, and then we will be able to back up our advancements with reports prepared by one more authoritative research body.

In general, we stick to what we have already said in one of our previous posts: Each research report should be taken with a sprinkle of healthy skepticism, with decision-makers relying on the results of as much independent tests as possible.