Budget aware method improves fine grained multimodal perception training
BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Multimodal AI models that look at both language and images often miss small but important details in pictures because they usually analyze the entire image at once. The authors present a new training method called BAS-OPD that teaches these models to focus on key parts of images by selectively asking for extra help only on the most useful examples. This saves a lot of time and computational resources during training while still making the models better at noticing fine details. Their experiments show the approach works well even with limited chances to ask the helper model for guidance.
What this means in practice
- •For computer vision engineers: Train multimodal perception models more efficiently by selectively querying teacher models only on valuable image regions, reducing training costs without sacrificing accuracy.
- •For multimodal ai developers: Develop AI systems that understand text and images with improved detail recognition by using budget-aware self-distillation methods during model training.
Authors
Zihan Chen, Hengguang Zhou, Yuan Kang, Yiming Zhang, Wenhui Fang, Zenghui Ding, Yining Sun, Cho-Jui Hsieh
Abstract
Multimodal large language models (MLLMs) often struggle with fine-grained visual perception when processing complete images, as critical evidence may only appear in local regions. On-policy self-distillation (OPD) enables transferring privileged visual knowledge from informative views to full-image policies, but querying the teacher for every rollout introduces substantial supervision costs. In this work, we propose BAS-OPD, a budget-aware selective OPD framework that allocates teacher supervision under limited query budgets. Instead of querying all rollouts, BAS-OPD selects informative samples while maintaining full-batch student generation. We explore random, uncertainty-based, and learned utility-based selection strategies, where the learned selector estimates query value from detached rollout statistics and online utility signals derived from student--teacher agreement and teacher confidence without additional student forward passes. BAS-OPD only changes training-time supervision allocation and preserves single-pass full-image inference. Experiments on fine-grained multimodal perception benchmarks demonstrate that BAS-OPD achieves strong performance while substantially reducing teacher supervision costs, highlighting the effectiveness of selective OPD under constrained budgets.