Reinforcement learning improves long reasoning with better probability rewards
Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
Artificial Intelligence
Summary
Training AI models to solve complex problems often involves giving them rewards based on how likely their answers are correct. This paper reveals that as these problem-solving steps get very long, the reward signals tend to become almost the same, making it hard for the AI to learn properly. The authors identify this issue, called Posterior Concentration Phenomenon, and introduce a new method that adjusts training to avoid it. Their approach helps AI models learn more efficiently and perform better on many reasoning tests.
What this means in practice
- •For ai development teams: Train language models more effectively on tasks requiring long decision sequences without relying on external answer checkers.
- •For automated reasoning developers: Improve stability and efficiency of AI systems tackling complex multi-step mathematical or logical problems using probability-based rewards.
Authors
Shiu-Hong Kao, Yubo Zhao, Zhenyu Tian, Pengzhan Sun, Yicong Li, Angela Yao
Abstract
Verifier-free reinforcement learning with probability-based rewards offers a promising way to train LLMs on general reasoning tasks where external verifiers are unavailable. Yet the reliability of these rewards, especially in long-horizon reasoning, remains underexplored. This work identifies a length-dependent failure mode of probability rewards, which we call the Posterior Concentration Phenomenon (PCP). We show that the probability of a reference answer conditioned on a reasoning trace often collapses to a low-variance interval as the trace becomes lengthy. This phenomenon results in nearly indistinguishable rewards, which, under GRPO-based settings, makes probability-based policy optimization unstable and inefficient. Motivated by this, we propose Reinforcement Learning with Concentration-aware Posterior Rewards (RLCPR), a verifier-free RL framework to explicitly account for PCP for better optimization stability and token efficiency. It has two components: uncertainty-aware data sampling, which reduces concentration-prone rollouts before generation, and concentration-aware regularization, which penalizes unnecessarily long traces when posterior rewards collapse. Extensive experiments show that, alongside higher token efficiency, RLCPR outperforms the state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks, including general-domain and mathematical reasoning challenges.