Improved visual detail recognition with evidence-aligned self-teaching method
Evidence-Aligned Multimodal On-Policy Self-Distillation for Fine-Grained Visual Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Fine-grained visual understanding means recognizing tiny details in complicated pictures, which is hard for computers. The authors studied a way where a smarter version of the model (teacher) guides a current version (student) by focusing on key visual clues. They improved this teaching by making sure the corrections match real evidence in the image, filtering out distractions caused by how the teacher sees the pictures. Their method keeps only the most helpful guidance, leading to better recognition of details than previous approaches.
What this means in practice
- •For computer vision engineers: Improve models that identify fine visual details by using evidence-aligned self-distillation to reduce irrelevant teacher corrections and boost accuracy.
- •For autonomous vehicle developers: Enhance object detection systems with better fine-grained visual cues alignment, leading to safer recognition of small but critical features in real-world environments.
Authors
Nanxing Hu, Qiwei Yan, Jinchao Zhang, Guoliang Kang
Abstract
Fine-grained visual understanding requires models to recognize small details within complex images. Multimodal on-policy self-distillation (OPSD) addresses this challenge by using a teacher conditioned on evidence-centered crops to supervise a student conditioned on original images along student-generated trajectories. Ideally, teacher corrections, the distributional changes from the student toward the privileged teacher, should be driven by task-relevant visual evidence. However, the designs that make the teacher effective also introduce other interference. Using a lagged or frozen teacher improves training stability but introduces a model-state gap from the evolving student, while cropping enhances task-relevant evidence but also loses the visual context. These two sources of interference make the teacher corrections not purely rely on the visual evidence. We introduce Evidence-Aligned multimodal on-policy self-Distillation (EAD), which retains the crop-conditioned teacher as the target but constructs a separate evidence reference for weighting the corrections. To exclude the effect of lagged model-state from this reference, EAD measures prediction changes using the current student. To avoid crop-induced context changes, EAD masks the evidence region in the original image while preserving the other visual context. The change from the student's masked-image prediction to its original-image prediction provides a controlled reference for the direction in which the visual evidence shifts the student's prediction. EAD weights each teacher correction by its cosine alignment with the reference, i.e., retaining aligned corrections and downweighting the rest. Retaining only 6\% of the supervision mass of dense OPSD, EAD consistently outperforms previous state-of-the-art methods.