AI struggles to spot incorrect but believable code reviews
CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?
Software EngineeringArtificial Intelligence
Summary
Code review comments are important for software quality, but sometimes AI-generated comments seem plausible while being technically wrong, which can mislead developers. The authors created a new benchmark called CRJudgeBench to test whether AI can tell if a code review comment is actually correct or just sounds convincing. They also built a system named Sentinel that checks the code repository to verify comments before judging them. Sentinel performs better than existing models but still only gets about three-quarters of cases right, showing this is a challenging task.
What this means in practice
- •For software development teams: Automatically evaluate if code review comments are truly correct to help developers trust and act on reviews more reliably.
- •For software quality assurance teams: Use AI tools to flag misleading or technically inaccurate code review comments when checking complex code changes.
Authors
Yue Pan, Jiawei Li, Ziyuan Zhang, Xiangxin Zhao, He Ye
Abstract
Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment's core technical claims are correct and applicable to the reviewed code in its repository context. Existing code-review benchmarks primarily evaluate review generation, issue discovery, or general comment quality, but do not directly assess whether an agent can determine the technical trustworthy of an individual review comment. To fill this gap, we introduce CRJudgeBench, a benchmark of 1199 instances constructed from real pull requests and expert-verified perturbations, covering both trustworthy and plausible but untrustworthy comments. We further present Sentinel, a repository-grounded agentic judge that actively gathers code evidence to verify review comments before making judgments. Starting from Qwen3-Coder-30B-A3B-Instruct, Sentinel is trained on the CRJudgeBench training split through iterative action-level learning from a privileged teacher. On the 359-instance CRJudgeBench test set, Sentinel achieves 76.60\% accuracy, outperforming GLM-5.3 by 6.13 percentage points and its base model by 19.78 points. These results show that even state-of-the-art general-purpose LLMs struggle to identify untrustworthy comments, while iterative action-level learning substantially improves the accuracy of repository-grounded trustworthiness judgments. Our dataset is available at https://huggingface.co/datasets/dcloud347/CRJudgeBenchmark