Olive improves language model training by learning from teacher continuations
Learning from Teacher Continuations at Student States
Computation and Language
Summary
Training smaller language models to learn from bigger ones can be tricky because learning from fixed examples or imperfect feedback makes the student model less reliable. The authors introduce OLIVE, a method where the student model writes part of a sentence, and then the teacher model continues it. This helps the student learn better by focusing on more relevant examples and feedback. OLIVE trains faster and performs better on challenging reasoning tasks, improving over earlier methods.
What this means in practice
- •For ai engineers: Train efficient language models that maintain reasoning skills using limited computing resources with OLIVE.
- •For chatbot developers: Improve chatbot performance on complex question answering by distilling knowledge from larger models more effectively.
Authors
Haojin Wang, Dylan Zhang, Huaibo Chen, Suhao Yu, Yihang Sun, Zhanyang Jin, Jiaying Ye, Dianqi Li, Prasanna Sattigeri, Kamal Youcef-Toumi, Hao Peng
Abstract
We present OLIVE (OnLine InterVEntion). At each iteration, the evolving student policy generates a new prefix, the teacher continues it autoregressively, and the student is updated using cross-entropy computed on the teacher-generated tokens. Each design choice targets a corresponding limitation of existing distillation methods: (1) sequential covariate shift in offline supervised fine-tuning (SFT) on fixed teacher trajectories, (2) fragmented supervision under prefix failure in token-level on-policy distillation (OPD), and (3) the need for access to teacher token probabilities in distribution-matching distillation. OLIVE achieves higher reasoning performance than OPD (with a top-16 KL approximation) at comparable GPU-hour cost. Our asynchronous implementation further reduces OLIVE's total training time by 23.8\%. We evaluate OLIVE on both hard reasoning tasks and agentic tasks which reflects modern post-training scenarios, and it consistently outperforms existing distillation methods under the same training budget. By regenerating prefixes from the evolving student, OLIVE continues improving after offline distillation plateaus while better preserving the general capabilities and plasticity of the student. Using only text from GPT-5.4-mini, continuously training with OLIVE outperforms offline SFT from the same teacher by 13\% on ScienceWorld. These results support OLIVE as an effective and efficient approach to online language-model distillation.