Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment
Software Engineering
Summary
The authors studied how automated code reviewers work when used directly in coding terminals before making official code changes. Instead of just looking at how well these tools perform, they explored the step-by-step actions the reviewers take. They found that these reviewers are more precise but often spend extra time exploring and double-checking code. Success happens when the reviewers plan better and do less rechecking later. Their work suggests future evaluations should consider the reviewers' behavior patterns and costs, not just accuracy.
Authors
Wachiraphan Charoenwet, Kla Tantithamthavorn, Patanamon Thongtanunam, Hong Yi Lin, Minwoo Jeong, Ming Wu
Abstract
Agentic code review in terminal-based environments enables early feedback during local development before pull request creation. However, existing evaluations remain performance-centric and fail to capture the dynamic behaviors of repository-grounded agentic reviewers. Understanding these behaviors is critical for identifying how agentic reviewers succeed, fail, and incur hidden operational costs in practice. Then, we analyze the reviewers' behavior based on their trajectories. Our results show that agentic reviewers achieve higher review precision but incur substantial exploration and validation overhead, while successful reviews are associated with stronger planning and less downstream validation. These findings highlight the potential benefits of trajectory-aware and cost-sensitive evaluation of future agentic code review systems.