Trust Region Policy Distillation
2026-07-06 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors show that trying to teach a new policy all at once can be very unstable, so they created a method called Trust Region Policy Distillation (TOP-D) that breaks the process into safer, smaller steps. They prove mathematically that TOP-D keeps the learning process stable and improves gradually without making training slower. When tested, their approach made training more reliable, faster, and better on math reasoning tasks compared to older methods. This new method does all that without needing extra computing power, making it a strong alternative to existing techniques.
Policy DistillationTrust RegionOn-Policy LearningGradient VarianceConvergence AnalysisSample EfficiencyTraining StabilityMathematical Reasoning Tasks
Authors
Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, Mao Yang
Abstract
Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.