SecProbe method improves testing for coding agents’ cybersecurity skills
SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
Cryptography and SecurityArtificial Intelligence
Summary
Checking how well computer programs recognize and fix security problems in code is hard because existing tests are fixed and don’t change as programs get better. The authors created SecProbe, a system that picks or generates coding tasks based on how well an AI is already doing, focusing on areas where more testing is most useful. They tested it with hundreds of tasks and found that current models still struggle to find and fix many security issues. SecProbe uses fewer tasks yet gives a similar understanding of an AI’s ability, making security testing more efficient.
What this means in practice
- •For software security teams: Benefit from more targeted and efficient testing of AI tools for detecting and fixing coding vulnerabilities across multiple programming languages.
- •For automated code review tool developers: Use adaptive task generation to improve evaluation workflows and better measure vulnerability detection and repair capabilities in coding assistant models.
Authors
Xiaonan Luo, Yue Huang, Kehan Guo, Ping He, Chuan Zou, Chujie Gao, Lichi Li, Yuchen Ma, Zhangchen Xu, Zichen Chen, Yufei Han, Xiangliang Zhang
Abstract
Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, \textsc{SecProbe} estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33\%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, \textsc{SecProbe} achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.