AI code review agents detect only a third of known defects
CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents
Software EngineeringMachine Learning
Summary
Checking computer code for mistakes is hard because understanding the code often needs extra context and judgment. The authors created a new way to test AI tools that review code by focusing on specific known bugs and their fixes in real projects. They measured both whether the AI found the bugs and how useful its suggestions were. Their tests showed that even the best AI code reviewers catch only about one-third of the problems, highlighting there’s still much room for improvement.
What this means in practice
- •For software development teams: Evaluate and compare AI tools for automated code review by measuring how well they detect known defects within real project contexts.
- •For ai tool developers: Use the CR-Bench dataset and metrics to improve AI models that assist in code review by focusing on precise defect detection and actionable feedback.
Authors
Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta
Abstract
The rapid growth of automated code generation has created a pressing need for automated code review. Yet code review resists full automation: it depends on context beyond the \textit{diff}, on the author's intent, and on judgments about design and maintainability on which even expert reviewers disagree. Existing code review benchmarks lack standardization and rely largely on noisy human review comments scored with text-similarity metrics, which reward surface overlap and fail to measure precision. We introduce \texttt{CR-Bench}, a defect-centric evaluation framework that recasts SWE-Bench issues as pull-request review tasks with full repository context. After filtering for review-detectable defects and grounding each to a single originating pull request, \texttt{CR-Bench} contains 584 instances, with a 174-instance \texttt{CR-Bench-Verified} subset derived from SWE-Bench Verified. Each instance pairs a pull request with a known defect, its ground-truth fix, and a taxonomy label for bug category, impact, and severity. We propose two complementary metrics: \emph{Known-Defect Detection Rate} (KDDR), which measures whether an agent identifies the labeled defect, and \emph{Usefulness Rate}, which measures the fraction of review comments that are actionable. We evaluate eight code review agents over four backbones, plus commercially available Gemini Code Assist, using three independent LLM judges. Even the strongest systems detect only about one-third of known defects (KDDR $\approx$ 33\%), and Gemini Code Assist detects 8.1\%.