Stable training improves language model knowledge transfer
TV-Regulated OPD: Direction Matters in On-Policy Distillation
Machine Learning
Summary
Training large language models to share knowledge with smaller ones can be unstable and noisy. The researchers found that focusing on the direction of learning signals, rather than their exact values, helps keep training steady. They developed a new method called TV-OPD that smooths out these signals, leading to more stable training and better results. Their experiments showed this method consistently improved performance and reduced fluctuations late in training.
Large language modelsOn-policy distillationTraining stabilityToken-level advantageTotal variationKnowledge transferMachine learningSignal smoothing
Authors
Han Xiao, Yifan Niu, Dongyi Liu, Chang Luo, Jia Li
Abstract
On-Policy Distillation (OPD) facilitates the transfer of knowledge from domain expert to student in the post-training phase of Large Language Models (LLMs). However, the supervision signals in mainstream OPD methods suffer from high variance and noise which is generally instable during training. In this work, we systematically investigated what really matters to the performance and the fundamental mechanisms behind the instability during training. We found that retaining only the sign of token-level advantages is sufficient to achieve the performance comparable to standard OPD. Meanwhile, smoother and bounded advantages can stabilize the training process without sacrificing its performance. These motivated us to shape the advantages using the Total Variation (TV) and propose a robust TV regulated On-Policy Distillation (TV-OPD) method. Benefiting from the bounded and diminished advantages, TV-OPD exhibits stable training dynamics and steady late-stage performance. We conducted comprehensive experiments and found that, across various settings, TV-OPD consistently achieved better performance and lower variance in the late-stage of training.