Stable reinforcement learning improves reasoning in large language models
Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification
Machine LearningComputer Vision and Pattern Recognition
Summary
Training large language models to think better using reinforcement learning can be tricky because the learning process sometimes becomes unstable or loses variety in its guesses. The authors propose two new methods to make this training steadier and more focused on useful examples. One method filters data to pick samples that help the model learn better, while the other organizes training steps to keep learning signals strong and prevent collapse. Together, these ideas help the model improve its reasoning on hard tasks more reliably and efficiently.
reinforcement learninglarge language modelstraining instabilityentropy collapserollout samplingdata filteringbatch stratificationgradient signalsfine-tuningmodel reasoning
Authors
Yimeng Ye, Shuang Chen, Wenxuan Huang, Manyuan Zhang, Kaituo Feng, Zhangquan Chen, Jiayu Chen, Yucheng Zhou, Yicheng Xiao, Zhiyuan Feng, Tianyu Shi
Abstract
While Reinforcement Learning (RL) effectively incentivizes reasoning in Large Language Models, current pipelines are hindered by training instability and rapid entropy collapse. These limitations often stem from "Rollout Silencing" and low-quality gradient signals in standard sampling procedures. In this work, we propose a robust, data-centric framework to stabilize RL training. We first introduce Potential-Aware Query Mining (PAQM), which filters data dynamically to focus on the "Distillation Zone"---samples with high potential for capability elicitation. Furthermore, we present Hybrid Stratified Replay (HSR), a novel mechanism that restructures batches by stratifying rollouts based on Path Entropy, a rollout-level confidence proxy, and outcome reward. Within each optimization step, HSR reuses current-policy "Stability Anchors" and "Hard Negatives" to construct high-contrast optimization groups, then clears its buffers before the next step. This approach mitigates entropy collapse while improving the utilization of learning signals under limited compute. Our method outperforms strong baselines on complex reasoning tasks, offering a principled solution for stable and efficient RL fine-tuning.