Language model learning improves with rewards for partial progress
Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching
Machine Learning
Summary
Training language models to complete long and complicated tasks is hard because feedback usually comes only at the very end, making learning slow. The authors studied a new way to give earlier, more frequent feedback by rewarding partial progress along the way, which helps models learn faster and better. They created a method called progressive point matching that compares parts of the model's work to a reference example to provide these in-between rewards without bias. Their method works much better on hard math problems than just giving rewards at the end.
What this means in practice
- •For language model developers: Train language models to solve complex multi-step tasks faster by using partial progress rewards based on comparison to reference solutions.
- •For math education software makers: Improve automatic math problem solvers to handle very challenging problems more effectively by rewarding stepwise progress during training.$Commercial implications: Enables selling advanced automated math tutoring or assessment tools that improve success on tough problems with step-level feedback.
Authors
Preston Fu, Kevin Frans, Oleh Rybkin, Sergey Levine, Aviral Kumar
Abstract
Current paradigms for training language models via reinforcement learning rely heavily on sparse outcome rewards. However, as we pursue tasks that require longer and more complicated trajectories, such strategies result in slow learning. Prior work has attempted to address this problem by rewarding partial progress; however, naive formulations are often biased and converge to suboptimal policies. We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks by rewarding partial progress on a segment level, both theoretically and empirically via synthetic environments. We then show how progressive point matching can be practically instantiated using a single reference trajectory per task. On extremely hard math reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time token budgets when measured by success rate or pass@k.