Agentic code review tools find many but miss tricky vulnerabilities
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
Software EngineeringArtificial IntelligenceCryptography and SecurityProgramming Languages
Summary
Finding security problems in software before code changes get added is hard. This paper introduces a new test called SecPRBench that uses real-world coding errors to check how well AI code reviewers find vulnerabilities. The tests show AI models catch between about 33% and 78% of bugs, with the best models doing better. Sometimes, when the AI misses a bug, it points to the right file but suggests the wrong fix, showing these are complex tasks.
What this means in practice
- •For software security teams: Identify which AI agents best improve catching real-world vulnerabilities before code merges in critical systems.
- •For software developers: Use insights about AI review strengths and weaknesses to prioritize manual checks on complex or subtle security flaws.
- •For cybersecurity service providers: Develop enhanced AI-driven security audit tools that combine different agentic models for improved vulnerability detection.$Commercial implications: Enables products offering automated pre-merge security code reviews with higher recall rates for clients.
Authors
Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang, Zhiheng Lyu, Lingming Zhang
Abstract
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.