Reinforcement learning improves efficiency by focusing exploration on key decisions
Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
Machine Learning
Summary
Reinforcement learning often requires generating many complete example runs, which can be costly. The authors propose a new way to figure out which points in these runs the learning model changes its mind most, so it can focus on exploring different options exactly there instead of everywhere. Their method, called Hindsight-Divergence Localization, reuses parts of previous runs and only generates new parts where important choices happen. This approach saves time and computing power while improving how well the model learns across multiple tasks.
What this means in practice
- •For machine learning engineers: Reduce computing costs and speed up training when developing reinforcement learning models for complex tasks.
- •For autonomous agent developers: Improve decision-making models in agents by focusing learning efforts on critical decision points identified through hindsight analysis.
Authors
Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao
Abstract
Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5$\times$ reduction in generated tokens and a 1.8$\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.