On-policy distillation can transfer teacher failures in vision-language models
Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Machine LearningComputer Vision and Pattern Recognition
Summary
This paper looks at how teaching a vision-language AI model by letting it learn from its own actions (on-policy distillation) compares to just learning from pre-recorded examples (supervised fine-tuning). The authors find that if the teacher AI model often fails on the student AI’s own examples, the student will also inherit those failures. This means that the method is only better when the teacher is reliable on the student’s paths. Filtering out incomplete teacher examples helps the student learn better. So, learning from the teacher’s own examples isn’t always better and can pass on mistakes.
What this means in practice
- •For ai model developers: Improve model training by choosing when to apply on-policy distillation or supervised fine-tuning based on teacher reliability.
- •For automated reasoning system builders: Design training pipelines that filter out incomplete teacher demonstrations to prevent propagating errors in vision-language reasoning tasks.
Authors
Frank F. Yang, Kefei Yan, Yiren Wang, Yiyang Wang
Abstract
Improving the reasoning capabilities of vision-language models (VLMs) often relies on post-training with supervision from stronger models. Two common strategies are supervised fine-tuning (SFT), which trains on teacher-generated demonstrations, and on-policy distillation (OPD), which provides teacher feedback on trajectories generated by the student itself. While OPD can be more data-efficient by supervising student-generated trajectories, it is unclear how the teacher's supervision at each state affects the student's learned behavior. To study this question, we conduct a controlled comparison of OPD and filtered SFT across data budgets and tasks where the teacher exhibits different levels of reliability, and analyze how these differences shape student accuracy and completion behavior. Across the two tasks, we find that OPD is effective only when the teacher remains reliable on student-generated trajectories. On ChartQA, where the teacher completes 99% of its solutions, OPD trained on only 10% of the data comes within 1.4 accuracy points of full-data SFT. In contrast, on Geometry3K, where the teacher fails to complete 22% of its solutions, OPD underperforms SFT by 7.9 points, with most of the gap explained by incomplete responses. More strikingly, starting OPD from an SFT student reduces its completion rate from 76% to 62%, effectively erasing the completion behavior learned through SFT. Filtering teacher demonstrations for completeness alone recovers much of SFT's advantage. These results show that on-policy supervision can transfer not only teacher knowledge but also persistent failure modes on student-visited states. Code is available at https://github.com/ffy208/opd-inherits-teacher-vlm.