Grasp enhances text image retrieval with interpretable concept matching
Concepts Complement Dense Semantics: Learning Compact Sparse Spaces for Text-Image Retrieval
Information RetrievalComputer Vision and Pattern Recognition
Summary
Finding matching images and texts that talk about the same thing is hard because general summaries can miss small but important details. The authors created GRASP, a new way to add clear and specific visual and textual clues, called concepts, to help find better matches. This adds a small, clear layer alongside usual methods, making it easier to understand why an image and text match. Their tests show this method works better than previous approaches that tried similar ideas.
What this means in practice
- •For search engine developers: Improve text-image search accuracy by integrating concept-based sparse matching to complement dense embeddings.$Commercial implications: Enables more precise multimedia search features for commercial search engines by providing interpretable concept grounding.
- •For digital marketing teams: Create better image tagging and retrieval systems for targeted advertising using concept-guided text-image matching.
Authors
Yoonseo Kim, Jungwoo Choi, Cheonyoung Park, Youngwook Kim, Yongho Song, SeongKu Kang
Abstract
Cross-modal retrieval has been advanced by vision-language pre-trained models that encode images and texts into a shared dense embedding space. While dense representations effectively capture overall semantic similarity, they often obscure fine-grained visual-textual information needed for precise cross-modal matching. Recent methods introduce a learned sparse branch to complement dense matching with lexical evidence, but they rely on a redundant language-model token space and lack explicit grounding for sparse dimensions. We propose GRASP, a compact and grounded sparse learning framework that mines visual-textual concepts from the corpus. A lightweight sparse head is trained to predict concepts relevant to each image or text, yielding interpretable concept-level evidence that complements dense semantic matching. Extensive experiments show that GRASP improves retrieval accuracy over the state-of-the-art dense-sparse baselines while yielding a more compact and grounded sparse space.