Model improves visual answer accuracy by learning from decision boundaries
BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation
Artificial Intelligence
Summary
It can be hard for AI that looks at pictures and answers questions to tell closely related answers apart, especially in special fields like medicine. The researchers designed a method called BIRD that finds tricky cases where the AI is unsure and learns exactly which visual clues help it choose the right answer. This approach teaches the AI with clearer explanations based on those clues, helping it make better decisions. Testing showed this method works better than similar ways to improve AI understanding in areas like medical images and charts.
What this means in practice
- •For medical imaging analysts: Improve AI models to interpret complex medical images by focusing on critical visual differences that matter for diagnostic answers.
- •For business intelligence teams: Enhance AI-driven chart and data visualization analysis tools by teaching models to discriminate subtle visual distinctions in graphs and reports.
Authors
Anglin Liu, Yanlin Wu, Ruichao Chen, Yuting Zhang, Qingyuan Zeng, Pengxiang Cai, Ziqi Gong, Muchen Li, Jintai Chen
Abstract
Adapting general-purpose multimodal large language models (MLLMs) to specialized domains requires learning domain-specific decision criteria, which often hinge on subtle visual distinctions between otherwise plausible answers. Rationale augmentation aims to expose such evidence through additional observations or inter-sample comparisons, yet a visually valid cue is not necessarily decision-relevant: it may describe how samples differ without changing the model's relative preference between competing answers. We therefore introduce BIRD, a self-improving Boundary-Informed Rationale Distillation framework that uses model-specific confusions to locate unresolved local decision boundaries and distills the evidence that resolves these confusions into rationales. For each sample, BIRD retrieves candidate neighbors from the target MLLM's own representation space and selects the most confusable one according to its answer preferences. It then generates answer-blind candidate evidence from their visual differences and functionally verifies which evidence most effectively strengthens the model's preference for the correct answer while avoiding inappropriate transfer across the pair. The verified evidence is then distilled into a single-sample rationale for standard supervised fine-tuning. Experiments on medical and chart VQA show that BIRD outperforms competing rationale-augmentation methods across two target MLLMs, while further analyses demonstrate clearer separation of confusable answers and stronger gains from model-matched supervision.