VETTA improves multi-turn LLM agent success by coordinating credit assignment
VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
Machine Learning
Summary
Multi-turn language agent systems often get sparse feedback after many exchanges, making it hard to know which part of their responses helped with success. The authors introduce VETTA, a method that figures out credit both at the level of whole responses and individual words generated, combining these to improve learning updates. They use a smaller critic model shared across tasks to keep computations efficient. Testing on two difficult benchmarks shows VETTA significantly improves agent task success while reducing computational cost.
What this means in practice
- •For llm system developers: Improve multi-turn language model agents’ performance by integrating token-level and turn-level feedback for better learning efficiency.
- •For ai product teams: Build more reliable interactive AI assistants by applying VETTA’s credit assignment method to increase task success rates.$Commercial implications: Enables more effective AI assistant products with higher task success by leveraging coordinated feedback mechanisms.
Authors
Jiaju Chen, Min Yang, Jinghua Piao, Xiaochong Lan, Xu Xia, Xiangnan He, Yong Li
Abstract
Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two related credit-assignment questions: which responses helped achieve the outcome, and which generation decisions mattered within each response? Existing methods typically focus on only one level: turn-level methods evaluate complete responses but do not distinguish the decisions within them; token-level methods can propagate feedback across turns but do not explicitly model credit for each response. These complementary limitations motivate learning credit at both levels and coordinating it in a single policy update. We introduce VETTA, a credit assignment method that jointly learns turn- and token-level values through separate heads on a shared lightweight critic. VETTA computes advantages along both temporal sequences and combines each turn advantage with a within-response-centered token residual for PPO updates. Furthermore, to reduce value-learning cost, the critic retains only early Transformer blocks from the pretrained checkpoint used to initialize the actor. On two challenging agent benchmarks, ALFWorld and WebShop, VETTA improves success rates over PPO by 37.5% and 22.3%, respectively, with Qwen2.5-1.5B-Instruct and achieves success rates of 95.5% and 76.0%, respectively, with Qwen2.5-7B-Instruct. Critic-depth comparisons further show strong task performance with substantially lower critic-side computation. These results suggest that a compact shared critic can coordinate turn- and token-level credit to improve agent performance while keeping value estimation efficient. Code is available at https://github.com/Jiaju-Chen/VETTA-official.