Teacher model choices strongly influence learning and forgetting in fine-tuned AI

OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

Machine Learning

Summary

Training language models using guidance from other AI models can either help them learn new tasks or cause them to forget previous skills. The authors created OPDArena to compare different training approaches and how the choice of teacher AI affects learning and retention. They found that when the teacher had been trained with reinforcement learning rewards, the student model improved on new tasks while keeping old abilities. But if the teacher was trained by standard supervised learning, the student model tended to lose some of its prior knowledge. Changes in the student model’s responses outside the training task area were linked to forgetting more than changes inside the task area.

What this means in practice

  • For machine learning engineers: Choose RL-trained teacher models to fine-tune language models with less forgetting of prior capabilities.
  • For ai product developers: Improve performance retention in customer-facing AI applications by selecting distillation methods that minimize capability loss.

Authors

Yuanze Xu, Tianzhu Ye, Hui Sun, Wenhui Wang, Li Dong, Furu Wei

Abstract

On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources. It jointly evaluates mathematical learning, general capability retention, and predictive distribution changes. Teacher choice strongly shapes retention: students of RLVR-trained teachers improve on the target task while retaining prior capabilities, whereas students of SFT-trained teachers incur greater capability loss. Across post-training methods, out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change. To understand how teacher choice affects these changes, we examine the additional information teachers provide relative to the student's pretrained model. Controlled teacher comparisons show that the task-specific preferences teachers acquire through post-training make student updates more concentrated in-domain.
View PDFOpen arXiv