Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
2026-07-06 • Information Retrieval
Information RetrievalArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of storing and scoring many image tokens for vision-language retrieval, which can be costly. They propose SaMer, a method that merges image tokens into a small set of representative groups while keeping important object details intact. SaMer only uses object info during training and does not need extra tools when used later. This approach reduces data size a lot and improves retrieval accuracy on common image-text datasets. Their results show that keeping object-level information is key for good and efficient multi-vector retrieval.
vision-language retrievalmulti-vector representationtoken merginglate interactionobject-aware compressionFlickr30KMSCOCOquery-selectable evidencephrase-level groundingprojection layer
Authors
Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang
Abstract
Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into $K$ representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With $K=64$, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by $16.09\times$, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.