Multimodal models improve fine perception with blinking teacher method
Teach with Blinks: Breaking Linguistic Momentum in On-Policy Distillation for Fine-Grained Multimodal Perception
Computer Vision and Pattern Recognition
Summary
Fine-grained recognition in AI models that understand both images and language can be slow and error-prone. The paper finds that when teaching a smaller model to imitate a larger 'teacher' model's understanding, the teacher might get misled by the student's mistakes. To fix this, the authors introduce a method where the teacher 'blinks'—sometimes paying full attention, sometimes ignoring parts of what the student predicts—to provide better guidance. This approach yields better accuracy using less training data and also improves performance on tasks the model wasn't specifically trained for.
What this means in practice
- •For multimodal ai engineers: Train fine-grained vision-language models with fewer images while improving accuracy using the blinking teacher method.
- •For computer vision system developers: Enhance visual understanding in AI systems across diverse tasks by integrating refined teacher supervision to reduce errors in perception.
Authors
Tangzheng Lian, Hang Zhang, Guanyu Hu, Renjiu Hu, Zhensong Zhang, Jiankang Deng, Jifei Song
Abstract
On-policy distillation (OPD) has emerged as an effective post-training approach to improving fine-grained perception in multimodal large language models by amortizing costly inference-time zooming operations into a single-pass recipe. However, vanilla OPD is inherently susceptible to linguistic momentum: the teacher can follow the student's erroneous prefixes, thus weakening corrective supervision. To address this, we propose Blink On-Policy Distillation (BlinkOPD), in which the teacher "blinks" at every token: it reads the same prefix once with its eyes open and once with its eyes closed. A first-glance gain further strengthens the correction on rollouts that conflict with the teacher. By extracting more effective supervision from each on-policy trajectory, BlinkOPD achieves state-of-the-art average performance across six fine-grained benchmarks using only 500 training images (12x less data) than existing OPD methods. Beyond fine-grained perception, BlinkOPD also demonstrates strong zero-shot improvements across ten held-out benchmarks spanning general visual perception, visual hallucination, and multimodal reasoning abilities. Code: https://github.com/Supltz/BlinkOPD