Multimodal retrieval method improves confidence guided answer updates
CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation
Computation and Language
Summary
Sometimes computers need extra information from text to answer questions about images more accurately. The authors found that current methods often bring back many repeated or not-so-helpful text snippets. They created CLIMB, which smartly selects a small set of different, relevant texts and only updates answers when it's more confident. This way, answers get better without extra training or changing the main parts of the system.
What this means in practice
- •For ai product developers: Integrate more reliable and concise evidence retrieval for image-based AI assistants answering complex questions.$Commercial implications: Enables selling enhanced multimodal AI assistants with improved answer reliability and efficiency.
- •For document search engineers: Improve search result diversity and confidence scoring to refine answers drawn from mixed text and image data.
Authors
Hang Gao, Wujiang Xu, Zhixing Zhang, Kai Mei, Jingyi Yang, Dimitris N. Metaxas
Abstract
Multimodal large language models (MLLMs) have shown strong visual reasoning abilities, but knowledge-intensive visual question answering often requires external textual evidence beyond the image and the model's parametric knowledge. Existing multimodal RAG systems commonly rely on Top-$K$ retrieval or reranking, which may return redundant passages and provide limited control over whether an answer update is sufficiently supported by the retrieved evidence. We propose \textit{CLIMB}, a training-free inference-time framework for multimodal RAG. CLIMB first constructs a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. It then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when the estimated confidence increases. This design provides a simple stopping criterion and reduces unnecessary refinement without modifying the underlying retriever or MLLM. Experiments on Encyclopedic-VQA and InfoSeek show that CLIMB consistently improves over retrieval-augmented multimodal baselines. Ablations further indicate that complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute to the final performance.