LLM value functions improve credit assignment in agentic reinforcement learning
LLM Value Functions for Credit Assignment in Agentic RL
Machine LearningArtificial Intelligence
Summary
RL agents using large language models struggle to learn from tasks where rewards come only at the end, making it hard to know which actions were good or bad. The authors introduce a method that uses the pretrained language model itself to estimate how good each step is, avoiding extra learning components. This approach helps the agent figure out which actions deserved credit, improving success on several challenging benchmarks. They also show a way to better teach agents by focusing on the most important actions identified by these value estimates.
What this means in practice
- •For software engineers developing ai agents: Improve learning speed and success rates of autonomous agents in tasks with delayed rewards by using LLM-based value estimation without separate critic models.
- •For machine learning platform teams: Integrate on-policy distillation guided by LLM value functions to produce more sample-efficient training pipelines for complex language model agents.
Authors
Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot
Abstract
Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.