Self distillation improves language model reasoning with verbal guidance
Self-Distillation with Verbal Regularization
Computation and Language
Summary
Large language models can learn from themselves by using a "teacher" version that has extra information to guide its answers. But if the teacher gets too confident too soon, it can stop exploring better answers. The authors introduce a method called verbal regularization that nudges the teacher to rethink and explore more, helping the "student" model learn better. This method improves accuracy and keeps the model’s answers less certain, which helps in problem solving.
What this means in practice
- •For language model engineers: Improve training of large language models by guiding teacher models to avoid overconfident shortcuts when learning complex tasks.
- •For software qa teams: Enhance automated code and problem-solving assistants by using self-distillation with verbal cues to increase answer reliability and exploration.
Authors
Jiazheng Li, Yawei Li, Junru Lu, Zhenyi Shen, Di Yin, Lin Gui, Xing Sun, David Rügamer, Xinpeng Wang, Yulan He
Abstract
On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.