Multimodal retrieval improves answer accuracy with generator feedback
Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment
Artificial IntelligenceInformation Retrieval
Summary
One problem with vision-language AI models that search for information to answer questions is that they often pick documents that seem relevant but don’t actually help produce the right answer. The authors propose a two-step method where the AI first imagines a text description from an image and question, then uses that to find documents. Afterward, the model learns to pick documents based on whether they helped it produce correct answers in the past, without needing human judgments. Their experiments show that this technique consistently finds better documents and improves answer quality on standard tests.
What this means in practice
- •For natural language processing teams: Enhance question answering systems combining images and text by improving how documents are selected to produce more accurate answers.
- •For multimodal ai developers: Refine multimodal retrieval workflows by incorporating generator-based feedback to better align document ranking with answer correctness.
Authors
Zhan-Lun Chang, Dong-Jun Han, Seyyedali Hosseinalipour, Mung Chiang, Christopher G. Brinton
Abstract
Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap: documents that appear relevant may not help the generator produce a correct answer. Motivated by this, we propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations. Our framework consists of two stages: in Stage 1, a VLM generates a hypothetical text passage from the image-query pair, which is used as the retrieval query for dense text search, bridging the image-to-text modality gap. In Stage 2, a cross-encoder reranker adapted with low-rank adaptation (LoRA) is fine-tuned using answer-supervised preference pairs mined from the frozen VLM: given the dataset answer label, a candidate document is labeled positive if the VLM produces the correct answer when given that document as context, and negative otherwise. This generator-guided signal is compatible with multiple alignment loss functions, including contrastive (triplet) loss, pairwise direct preference optimization (DPO), and supervised fine-tuning (SFT), and supports periodic re-mining to refresh preference pairs as the reranker improves. Experiments on VQA-X and A-OKVQA with Qwen3.5-2B and Qwen3-VL-4B-Instruct show that our proposed framework consistently outperforms rank-order, random, and REPLUG-style likelihood baselines under various alignment losses and pool size settings, suggesting that answer-level generator feedback is an effective supervision signal for preference alignment.