Reinforcement learning method improves token-level credit assignment
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Machine LearningArtificial Intelligence
Summary
Assigning rewards to individual steps in a task is tricky when training language models. The authors propose a method called HP-RLVR that uses a single attempt's future outcome to better figure out which parts helped or hurt performance. This approach needs fewer attempts per problem and trains faster than previous methods, while still keeping similar accuracy. It works by comparing how likely a model thought a token was before and after seeing the final outcome.
What this means in practice
- •For language model engineers: Train language models to learn from single-answer feedback more efficiently by better assigning credit to each generated token.
- •For autonomous agent developers: Improve reinforcement learning for agents handling sequential decisions without needing multiple attempts per scenario.
Authors
Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
Abstract
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.