CrowdStrike and Meta are jointly introducing CyberSOCEval, a new suite of open source benchmarks to evaluate the performance of AI systems in security operations. The collaboration aims to help organizations select more effective AI tools for their Security Operations Center.
Meta and CrowdStrike are addressing a growing challenge by introducing CyberSOCEval, a suite of benchmarks that help define what effective AI looks like for cyber defense. The system is built on Meta’s open source CyberSecEval framework and CrowdStrike’s frontline threat intelligence.
CyberSOCEval evaluates LLMs across critical security workflows such as incident response, malware analysis, and threat analysis comprehension. By testing AI systems against a combination of real attacker tactics and expert-designed security reasoning scenarios, organizations can validate performance under pressure and prove operational readiness.
Challenge for cyber defenders
Cyber defenders face an overwhelming challenge due to the influx of security alerts and evolving threats. To stay ahead of adversaries, organizations must embrace the latest AI technologies. However, many security teams are still in the early stages of their AI journey, especially when it comes to using Large Language Models (LLMs) to automate tasks and increase efficiency in security operations.
Without clear benchmarks, it is difficult to determine which systems, use cases, and performance standards actually offer an AI advantage against real attacks.
Practical application
These benchmarks allow security teams to pinpoint where AI delivers maximum value. Model developers gain a target for improving capabilities that increase ROI and SOC effectiveness.
The CyberSOCEval open source benchmark suite is now available to the AI and security community for evaluating model capabilities. Access to the benchmarks is available through Meta’s CyberSecEval framework.
These new benchmarks set a new standard for how AI should be built and deployed in the SOC, enabling defenders to stay ahead of adversaries.
Tip: CrowdStrike Falcon XDR now integrates with Rubrik Security Cloud