Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG
2026-07-12 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study how systems that answer questions using both text and images can struggle when images or data are unreliable or misleading. They create a new test called QIMG-7 to measure how often bad images cause errors in answers. They find that simply mixing all information together often leads to mistakes, so they suggest a method called source-aware trust resolution (SATR) that decides which answers to trust based on the reliability of the sources. Their approach improves accuracy compared to fully trusting all data. Overall, the authors show it is better to carefully choose what to trust instead of blindly combining all extra information.
Multimodal Retrieval-Augmented GenerationQuestion AnsweringImage AttacksData PollutionSource ReliabilityText-Only ModelsModel FusionTrust ResolutionFactual QABenchmarking
Authors
Saadeldine Eletter, Owais Aijaz, Preslav Nakov
Abstract
Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at https://github.com/SaadElDine/Trust_Before_Fusion.