Boosting LLM Exploration via Weak-Model Guidance in RLVR
2026-08-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study a problem in training language models to solve reasoning tasks, where models get too confident and lose diversity in their answers. They propose a new method that makes the model explore different reasoning paths by starting from partial answers generated by smaller models. This approach helps the main model avoid repeated similar reasoning and improves performance on math benchmarks, especially when looking for many correct answers. Their method works well without needing complicated extra training or reward setups.
Reinforcement LearningLarge Language ModelsPolicy EntropyExplorationReasoning CoverageReward DesignMathematical BenchmarksFine-tuningGenerative Diversity
Authors
Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.