Self evolving policy improves reasoning in large language models
Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation
Machine LearningComputation and Language
Summary
Large language models struggle to improve their reasoning when they get stuck exploring only wrong answers. The authors propose a method called Self-Evolution Policy Optimization (SEPO) that helps the model explore better answers by using its own correct reasoning examples as a guide. This approach lets the model repeatedly explore and learn from itself, improving steadily even on hard problems or with smaller models. Their experiments show SEPO outperforms other methods in various reasoning tasks.
What this means in practice
- •For ai developers: Build language model training pipelines that continuously improve reasoning capabilities without external demonstrations.
- •For automated reasoning teams: Improve the efficiency and robustness of logic and scientific reasoning systems with stable policy self-improvement cycles.
Authors
Wenhang Shi, Yiren Chen, Zhe Zhao, Jinhao Dong, Wei Lu, Xiaoyong Du
Abstract
Reinforcement learning with verifiable rewards (RLVR) enables large language models (LLMs) to continuously self-improve their reasoning capabilities. Compared to RLVR's scalar rewards, recent self-supervised on-policy distillation methods leverage denser supervision signals. However, both paradigms suffer from exploration bottlenecks, struggling to escape local optima when the policy fails to sample valid solutions. Relying on external demonstrations can bypass this issue; however, the off-policy nature hinders genuine self-improvement and the mismatched-distribution reasoning can not be fully exploited. In this work, we propose Self-Evolution Policy Optimization (SEPO), a framework that drives continuous reasoning capability evolution via guided exploration and in-distribution exploitation. Specifically, SEPO utilizes labels to elicit correct reasoning from the policy, unlocking the exploration potential. By treating current policy conditioned on this in-distribution reasoning as a self-teacher, SEPO distills the newly discovered solution back into the policy. Through this iterative cycle of policy-intrinsic exploration and exploitation, SEPO achieves robust and continuous capability evolution. Across scientific, tool-use, and complex logical reasoning domains, SEPO consistently achieves the best performance under identical training time or steps, demonstrating superior efficiency and convergence ceilings. Notably, SEPO maintains stable improvements even in scenarios where baselines severely underperform or fail entirely, such as on complex tasks or with small-scale models. Our code is available at https://github.com/Eric8932/SEPO.