Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
2026-07-16 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors studied AI security agents by looking not just at how well they succeed, but how much computational and tool-use 'cost' they incur during tasks. They tested these models on both offensive challenges (like hacking simulations) and defensive tasks (investigating security alerts). They found that offensive performance generally improves with more compute, while defensive success depends more on smart use of tools and data navigation rather than just more reasoning power. The authors suggest future evaluations should consider both task success and cost to better reflect real-world usefulness of security AI.
security-agentinference budgetoffensive securitydefensive securitycybersecurity benchmarkingCTF (Capture The Flag)SOC (Security Operations Center)telemetrytool use efficiencyeconomic efficiency
Authors
Paul Kassianik, Blaine Nelson, Yaron Singer
Abstract
Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.