Instruction-preserving decoding improves reliability of visual question answering
Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Visual question answering systems sometimes struggle to say "I don't know" when appropriate and instead give wrong answers. The authors found that previous methods to improve answers accidentally removed these honest abstentions. They created a new decoding approach that keeps the system's ability to abstain while still improving answer accuracy. This new method works better across many models and datasets, balancing correct answers with sensible abstentions.
What this means in practice
- •For visual ai developers: Create visual question answering systems that better handle uncertainty by appropriately abstaining when answers are unsupported.
- •For product teams for assistive devices: Enhance assistive visual tools by improving answer reliability and reducing incorrect responses in image-based queries.
Authors
Haonan Zhang, Ting Jin, Wenzhong Tang, Shuai Wang
Abstract
Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers. We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted. We therefore propose instruction-preserving decoding, which keeps the effect of the abstention instruction on the original-image prediction intact while leaving the contrast strength freely adjustable. Using a perturbed image and an image-free prediction as references, we obtain IP-VCD and IP-M3ID. On decision utility (DU), which credits both correct answers and supported abstentions, both methods outperform all compared non-IP baselines on every one of nine models on VizWiz, and IP-VCD also leads on eight of nine models on Food-101. Candidate measurements and paired per-question comparisons show that computing the instruction effect and the visual contrast term separately lets contrastive decoding support both correct answers and supported abstentions.