Teacher model helps fix student mistakes in multimodal reasoning
OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
When AI models try to describe images while thinking step-by-step, early mistakes can cause both the teacher and student models to make the same wrong guess, making it hard to fix. The authors found that although the student and teacher agree on the wrong answer, the teacher still prefers the correct interpretation when seeing the real image. They created a method called OPD-Aha that focuses on this hidden teacher preference to help the student rethink and correct itself mid-answer, improving accuracy in tasks that require understanding both images and text.
What this means in practice
- •For multimodal ai developers: Improve AI systems to self-correct reasoning about images while generating text responses, enhancing accuracy in image captioning and visual question answering.
- •For interactive assistant creators: Build assistants that can detect and pause to rethink flawed interpretations during conversation involving images for better user guidance.
Authors
Chenhao Qiu, Dawei Li, Yechao Zhang, Lei Gong, Zhen Tan
Abstract
Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence. Both models score these trajectories while conditioning on the same student-generated prefix. When a student misinterprets an image early in a response, this accumulating erroneous rationale eventually pulls the teacher away from its visual evidence. The teacher and student converge on the same hallucination, causing standard cross-model supervision to collapse precisely where correction is most needed. We find that the teacher's visual corrective preference is not lost under this misleading agreement. Comparing the predictions of the identical teacher given the real image and a visual null reveals that the privileged evidence still pushes the model toward the correct interpretation. We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy. This reconstructed target aggressively suppresses continuations that contradict the image. Trained with this objective, students learn to naturally interrupt their own flawed reasoning with reflection tokens such as wait and actually. After reflection, subsequent generation relies less on the accumulated erroneous text and more on the visual evidence. Correcting these trajectories mid-generation fundamentally alters the reasoning process, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks. Our code and models are available at https://github.com/Echochef/OPD-Aha.