On-policy distillation improves reasoning skills without new facts
On-Policy Distillation Teaches New Skills but Not New Knowledge
Computation and Language
Summary
Sometimes, we want computer programs that can think through problems step by step. This paper shows that a method called on-policy distillation helps models get better at reasoning in complicated ways but doesn't actually teach them new facts. The authors tested this by controlling what the teacher model knew and found that students mainly learn how to combine what they already know rather than gain new knowledge. So, this technique improves how models use facts they have instead of increasing what they know.
What this means in practice
- •For language model developers: Improve reasoning performance of language models by training them to better organize known facts instead of relying on new factual learning.
- •For automated math problem solvers: Enhance multi-step reasoning abilities in math models without needing to expand their stored factual database.
Authors
Yixuan Tang, Yi Yang
Abstract
On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses.