OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review
2026-08-10 • Software Engineering
Software Engineering
AI summaryⓘ
The authors identify problems in current AI-based code review tools, including inconsistent results and limited context when reviewing code changes. They propose OpenCodeReview, which uses fixed rules and controlled exploration to make reviews more stable and thorough across multiple files. Additionally, they introduce a separate checking step to catch incorrect AI comments, improving trust and accuracy. Their system performs better than existing tools on a benchmark with real code reviews and uses fewer computing resources.
LLMcode reviewnon-determinismcontext localitymultifile pull requestReAct loophallucinationfalsification filterSEM-F1SubAgents
Authors
Zhengfeng Li, Lei Zhang, Xianwei Wu, Zhengqi Zhuang, Yingjie Xu, Boge Wang, Shaofei Zhu, Chuan Wang, Guoping Rong
Abstract
LLM-based code review agents promise scalable, always-on review, yet current systems suffer from two intertwined weaknesses: (1) non-determinism--unbounded tool use makes review outcomes unstable, and (2) context locality--the reviewer's access remains bounded to the diff, capping discoverable issue depth. Both give rise to three challenges: misaligned context retrieval, a coherence-efficiency trade-off in multi-file pull requests, and hallucinated comments that erode trust. To address these, we introduce OpenCodeReview, built on deterministic engineering for uncertain agents: rather than granting maximal freedom, we inject determinism at three deliberate pipeline points. Rule-Guided Dispatch uses a multi-layer rule system to deterministically select files and review criteria, eliminating variability of agent-driven triage. Grounded File Review replaces free-form exploration with a curated tool set exposed through a ReAct loop, while file-level parallel SubAgents balance context coherence against efficiency and recover cross-file dependencies on demand. Independent Reflection introduces a falsification-first filter under an asymmetric information boundary--the reflector sees only the diff, not the agent's tool-augmented exploration--removing hallucinated comments without self-reinforcing bias, improving precision while preserving recall. On AACR-Bench (200 real-world PRs, 10 languages, 1,505 expert-verified comments), OpenCodeReview outperforms mainstream coding agents (e.g., Claude Code and Codex) across six LLM backends, achieving up to 2.17x higher SEM-F1 (25.10% vs. 11.57%) while consuming 5-15x fewer tokens. We open-source OpenCodeReview at https://github.com/alibaba/open-code-review.