Asymmetric self-distillation improves policy learning efficiency in large models
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Machine LearningArtificial IntelligenceComputation and Language
Summary
Training large language models to improve their decision-making can be tricky because copying themselves too closely might keep mistakes or lose good behaviors. The authors propose a new way called APSD that mixes the model’s own choices with its previous best answers differently depending on correct or incorrect actions. This method helps keep the good parts while fixing mistakes more strongly, speeding up learning. They tested this approach and found it works better and faster than some earlier methods.
What this means in practice
- •For ai model engineers: Build more efficient and reliable large language model fine-tuning with fewer training steps using APSD for improved policy optimization.
- •For machine learning platform teams: Enhance reinforcement learning workflows by integrating correctness-dependent target distributions to better balance good and bad policy updates.
Authors
Suwan Yoon, Seongryong Jung, Changhee Lee
Abstract
Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.