Canopy compresses multimodal evidence adaptively for better question answering
CANOPY: Adaptive-Granularity Evidence Compression for Multimodal RAG
Information Retrieval
Summary
When answering questions using many types of information like text, tables, images, and videos, picking the right amount of details to keep is tricky. Keeping too much can include irrelevant parts, while keeping too little can lose important context. The authors developed Canopy, a method that breaks down information into parts and picks the best-sized pieces to keep for each type without needing expensive large language model calls. It also knows when more information should be retrieved. This approach improves accuracy on various question-answering tasks while reducing the amount of information needed for the final answer.
What this means in practice
- •For enterprise search teams: Improve efficiency and accuracy of systems retrieving mixed media evidence for complex queries by selectively compressing relevant information with adaptive granularity.
- •For video analysis developers: Enhance retrieval tools that handle video alongside text and tables by dynamically compressing regions to reduce data while preserving needed context for downstream models.
Authors
Hyojeong Yun, Jueun Kim, Wook-Shin Han
Abstract
Multimodal RAG retrieves text, tables, images, and videos, but choosing a retrieval granularity does not determine how much context to retain within each item. Coarse units include irrelevant content, while uniformly fine selection can remove context needed to interpret the evidence. Existing compressors address this trade-off with modality-specific mechanisms, leaving open a shared procedure for adapting the retained extent region by region across heterogeneous items. We introduce CANOPY (Canonical Projection over Hierarchy), a framework for adaptive-granularity post-retrieval evidence compression. CANOPY represents retrieved items as hierarchies and uses a node encoder fine-tuned on gold evidence to score regions against the query. Parent-relative refinement compares these scores to select multiple regions at different granularities without LLM calls for node-level pruning. Because compression cannot recover evidence that was never retrieved, a critic requests targeted follow-up retrieval when it judges the accumulated evidence insufficient; newly retrieved items are compressed before being added. Across five QA benchmarks over a 33M-item heterogeneous corpus, CANOPY achieves higher average answer accuracy than the evaluated retrieval baselines. Ablations indicate that additional retrieval drives the main accuracy gains on multi-hop QA. In the unrouted Qwen3-VL-8B-Instruct setting, compression reduces reader-input evidence tokens by 14.2-27.7% relative to the same iterative pipeline without compression, with comparable answer accuracy.