Self instructing policy optimization improves reinforcement learning for language models
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Artificial IntelligenceComputation and Language
Summary
Large language models often learn by getting rewards after finishing a task, but this doesn't explain which words helped or hurt along the way. The authors of this paper created a method called SIPO that teaches the model to learn from its own guesses by comparing good and bad answers side-by-side. This method gives detailed feedback on each word, making it easier for the model to improve. Their experiments show SIPO works better than previous ways to teach these models without needing an outside teacher.
What this means in practice
- •For ai development teams: Improve training of language models for better reasoning and code generation without external teachers.
- •For software engineering teams: Build smarter code generation tools that learn from detailed feedback on intermediate steps.
Authors
Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.