What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels
2026-08-24 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how to better evaluate coding agents by focusing on how they perform tasks step-by-step, not just if they succeed. They found that current methods mix up different evaluation goals, making it unclear what is actually measured. To fix this, they created a new framework and a tool called SCAE to analyze which specific steps really cause actions in the coding process. Their experiments showed that actions depend more on past execution steps than on code structure, and that uncertainty happens more across tasks than at individual steps. They also discovered that some evaluation methods might be biased and measure relevance instead of true causal impact.
coding agentsprocess evaluationcausal attributionstructural causal modelexecution provenancetask uncertaintystep attributionreplay-based estimationcollider biascode-graph transitions
Authors
Jiawei He, Mengyu Shi, Jie jia, Xikai Yang, Dong Sun
Abstract
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.