In MITRE’s annual evaluation of security solutions, SentinelOne Singularity put up a perfect score for the fifth year in a row. Not only that, it also has an excellent signal-to-noise ratio. The latter is critical in a world where there the number of attacks only increases and taking action is required more frequently.
The MITRE ATT&CK Evaluations are a household name in the security world. Every year, security vendors participate in this real-world test to have their solutions tested. The organization behind this evaluation comes up with a different test each year, even though there is obviously some overlap between tests from different years. This year it focused on two major threats: widely deployed ransomware that targets Windows and Linux and modular malware that penetrates MacOS endpoints in multiple steps. These are real-world attacks simulated by MITRE. The idea is that the response of security solutions to them will give a good idea of their capabilities.
Also read: How do you interpret the results of the MITRE ATT&CK evaluations?
In the results of the tests, there are several things that matter. First is the accuracy of detection. Not only that, solutions must do so as quickly as possible. Furthermore, it is important that solutions detect the techniques that attackers use. Finally, MITRE would also like to see as few unnecessary alerts (i.e. noise) as possible. If SOC employees get these too often, they start to suffer from alert fatigue. This is disastrous, because then they also start missing real alerts (i.e. signals).
It is worth mentioning that this year MITRE is also working with false positives. These are alerts of things that may look like a real threat, but on closer inspection are not. Finally, compared to previous years, this year MITRE has taken the complete evaluation into its own hands. Previously, vendors ran the test on their own platforms. This year, MITRE analysts are taking care of that. That way, not only do the threats resemble those in the real world, but also the environment in which the solutions run.
SentinelOne sets perfect score (again) for detections
A 100 percent score in detections is nothing special for SentinelOne. That is, it has done so for the past five years. This year is no exception. SentinelOne’s Singularity platform detected all 16 attacks and all 80 steps that are part of those attacks. In addition, there was no delay in these detections. That is, the platform detected everything in real time, thanks in part to its built-in AI capabilities. There was also a perfect score in terms of detecting attack techniques. If you detect all the individual steps of an attack, you probably pick up the techniques right away as well.
The above results are obviously important. Meanwhile, however, it is also increasingly about how many alerts a security solution provides. Especially with complex attacks, it is nice if there is not a separate alert for every small component. Rather, it makes sense for a security solution to collect all of them and present them as a single alert. In addition, of course, the goal is also to have as few false positives as possible. In other words, the signal-to-noise ratio should be as good as possible.
Signal-to-noise ratio
In terms of signal-to-noise ratio, SentinelOne is also doing an excellent job, based on the figures we now have at our disposal. In fact, the Singularity platform produces 88 percent fewer alerts than the median. The differences at this point are very large among the security platforms and solutions that MITRE tested. Where SentinelOne has a total of 71 notifications, at the other end of the spectrum we see Qualys, for example, with 981,436 notifications.
We do not currently have the full dataset of the tests available, so what the exact implications of the above numbers are may not be entirely clear. At least SentinelOne seems to be right on target with its combination of 100 percent detection and excellent signal-to-noise ratio. A tremendously high value indicates that at least a lot of noise reaches the SOC analysts. By itself, it says nothing about accuracy in terms of detections. Those can all be in three, too, of course. The fact is, though, that a SOC analyst is a lot less likely to pick them out of a huge number of alerts than they would be when the signal-to-noise ratio is at the level of SentinelOne’s Singularity platform.