Language agents learn which actions lead to success faster
From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents
Machine Learning
Summary
Teaching language-based AI agents to complete long tasks is hard because they only get feedback at the end, not for each step. The authors propose SCALE, a method that helps the AI figure out how each step changes the chance of eventually succeeding. This extra insight helps the AI assign credit more reliably and improve its performance. When tested on several tasks and models, SCALE led to better success rates and more stable learning.
What this means in practice
- •For ai developers: Build language agents that better learn which intermediate actions increase chances of task success in complex tasks.
- •For robotics programmers: Improve robot controllers that use language instructions by assigning better credit to intermediate steps toward goal completion.
Authors
Jiaxi Liu, Shixiong Kai, Zhentao Tang, Yang Li, Yuqi Cui, Mingxuan Yuan, Junchi Yan
Abstract
Reinforcement learning for long-horizon language agents requires identifying whether intermediate actions make eventual task success more or less likely when supervision is available only at trajectory level. The main difficulty is that a critic with aggregate return-prediction accuracy may still fail to resolve the local value differences needed to assign credit to actions, especially under sparse feedback and function approximation. Conventional critics estimate future return, but policy improvement also depends on knowing whether each intermediate decision makes task success more or less likely. We introduce SCALE, an actor-critic framework that augments conventional return estimation with explicit prediction of eventual task success. At each interaction prefix, the critic predicts a return value and a Success Value, defined as the probability that the task will eventually be completed from that prefix. SCALE converts changes in Success Value into multi-step action credit through Success-GAE and modulates its contribution according to the estimated reliability of the predictor, while retaining the token-level PPO objective. Across five agent benchmarks and three model backbones, SCALE improves task performance over the compared RL baselines in most settings, with gains of up to 4.1 percentage points in success rate. Separate critic-learning analyses show that SCALE also improves critic estimation and training stability.