Visual retrieval system improves answers by selecting and organizing images
Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation
Artificial Intelligence
Summary
Sometimes, finding the right pictures to answer questions about complex documents is hard because important information is rare or scattered. The authors designed a system called SCoRE that carefully chooses only the relevant images while exploring and then organizes them clearly before giving an answer. This makes the answers more reliable and easier to understand because they come directly from chosen pictures instead of confusing notes. The system learns by practicing with example question paths and rewards that encourage finding good evidence and clear explanations.
What this means in practice
- •For document analysis teams: Generate accurate answers from complex visual documents by selecting and organizing relevant images before reasoning.
- •For ai system developers: Build visual question answering agents that maintain strict links between claims and original images for transparent reasoning.
Authors
Yucheng Shen, Lingyong Yan, Jiulong Wu, Shuaiqiang Wang, Jianmin WU, Dawei Yin, Min Cao
Abstract
Visual Retrieval-Augmented Generation (VRAG) empowers models to navigate and answer queries about visually rich documents by retrieving relevant page images as visual evidence and reasoning over their content. However, effectively utilizing this visual evidence is usually impeded by two main challenges. First, answer-relevant evidence is sparse and may be concentrated in a small region of one page or dispersed across multiple pages. Second, existing agentic methods often generate answers based on raw exploration trajectories or compressed textual memories rather than an explicitly organized set of supporting images, making answers susceptible to exploration noise and obscuring the evidence-backed reasoning trace. We argue that the bottleneck lies not only in evidence discovery but also in its preservation and organization before answer generation. We propose SCoRE (Selection and Consolidation for Robust Evidence), a unified agent loop for explicit evidence selection and consolidation. During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger, preserving earlier evidence while keeping the visual context bounded. At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence. This decouples final reasoning from exploratory trial-and-error while ensuring strict visual grounding via indexed claim-to-image linkages. To enable end-to-end optimization of this unified rollout, our training paradigm combines filtered cold-start trajectory distillation with evidence-aware reinforcement learning, whose reward promotes evidence coverage, consolidation compactness, and answer correctness.