Medical visual question answering improves with multimodal evidence retrieval
Enhancing RAG-based Medical VQA with Gated Multimodal Representations
Computer Vision and Pattern RecognitionComputation and Language
Summary
Medical AI systems that answer questions about medical images sometimes give answers that are not fully accurate. The authors developed a way for these systems to better find relevant medical evidence by combining information from both the image and the question together. Their method uses a special unit that merges text and image features, improving how the system searches for helpful documents. Tests show this approach works better than methods relying on just one type of input.
What this means in practice
- •For medical ai developers: Build medical question-answering tools that retrieve relevant clinical information using combined image and text queries for more accurate answers.
- •For healthcare it teams: Enhance clinical decision support systems by improving how medical images and related questions jointly guide information retrieval.
Authors
Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-Farías
Abstract
Medical Vision-Language Models (Med-LVLMs) have shown strong performance in Medical Visual Question Answering (MVQA), yet they may still produce factually inconsistent responses. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence; however, retrieval is often guided by a single modality, potentially overlooking the joint semantics of medical images and clinical questions. We introduce a multimodal RAG framework that learns a joint query representation through a Gated Multimodal Unit, enabling evidence retrieval to leverage information from both modalities while keeping the underlying Med-LVLM frozen. We evaluate the proposed approach on benchmark datasets. Results show that the multimodal query representation consistently outperforms a reference Med-LVLM and remains competitive with state-of-the-art methods. Our findings demonstrate the potential of multimodal query representations for more effective evidence retrieval in Medical VQA.