Concise captions and re-ranking improve image retrieval accuracy
Less is More: Concise Retrieval and Critic-based Re-ranking for Composed Image Retrieval
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Finding pictures that match changes described by text can be tricky when the descriptions include too much extra detail. The authors created a method that first makes short, focused descriptions emphasizing only the important changes. Then, they check the top pictures again to make sure they really fit the description and the original image. This two-step approach helps computers find the right pictures more accurately without needing new training.
What this means in practice
- •For digital content platforms: Improve user searches for modified images by filtering out noisy captions and ranking results more precisely based on changes described in text.
- •For e-commerce product teams: Enable better matching of product images with user queries that combine example images and modification instructions to find alternative versions.
Authors
Yuqian Zheng, Hyoju Ahn, Hyeongjin Ahn, Eunil Park
Abstract
Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image based on a reference image and modification text. Recent approaches commonly rely on large vision-language models to generate target image captions for retrieval. However, these captions may include redundant semantic details, introducing noise when retrieving the target image and leading to suboptimal retrieval results. To address this issue, we propose Less-is-More CIR, a two-stage training-free framework that combines Concise Retrieval and Critic Re-ranking. In the first stage, concise target-oriented captions are generated that preserve only the most discriminative modification semantics, thereby reducing semantic noise in the retrieval. In the second stage, a critic re-rank module further evaluates the retrieved candidates from the perspectives of modification consistency and reference preservation. Extensive experiments demonstrate the effectiveness and robustness of the proposed framework for the ZS-CIR. Our code is available at: https://github.com/dxlabskku/LM-CIR.