Visual distillation improves token-level reasoning in vision language models
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
Artificial Intelligence
Summary
Understanding images and language together is hard because it's difficult to tell which parts of a description really depend on the picture. The authors studied a way to focus training on parts of the response that actually relate to the image content, not just generic phrases. They do this by comparing how a model responds to the real image versus a changed image lacking key details. This helps the model learn better which words are important for reasoning about what is shown in the picture.
What this means in practice
- •For vision-language model developers: Improve training of image-language reasoning systems by focusing supervision on tokens grounded in specific visual evidence.
- •For multimodal ai engineers: Produce more accurate and interpretable vision-language models by softly weighting token importance during training using counterfactual imagery.
Authors
Jiahe Feng
Abstract
GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.