AI summaryⓘ
The authors work on ways to evaluate how well AI agents perform step-by-step during tasks, which is usually hard when tasks are long and uncertain. They show that by using reinforcement learning (RL) after training, you can create a useful scoring system without extra annotation or training. This scoring, called the progress advantage, compares the improved policy to the original and reflects how much better the agent is doing at each step. They test this method on various tasks and models, finding it works better than traditional confidence scores and even some dedicated reward models, all without needing special training. The authors also explore the properties of this method to help others use it in real-world AI systems.
reinforcement learningreward modelMarkov decision processpolicyadvantage functionstep-level evaluationlog-probability ratioagentic systemsuncertainty quantification
Authors
Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick, Sharon Li
Abstract
Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.