New method improves image and long text matching by balancing key and background details
Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
Artificial Intelligence
Summary
Matching images to long descriptions can be tricky because models often focus too much on the most obvious parts and ignore the less noticeable but still important details. This causes problems when those obvious parts change or disappear. The authors found this issue, which they call saliency collapse, and created a method named RoSA that teaches models to pay attention to both the standout and background information. RoSA uses a clever way to split images and text into important and contextual parts and aligns these parts together. This results in better match accuracy and more reliable performance even when obvious cues are missing.
What this means in practice
- •For search engine teams: Improve accuracy and robustness of image search results by better matching images to detailed user queries even when key visual cues are missing.
- •For e-commerce platform developers: Enhance product retrieval by aligning product images with long textual descriptions that include both prominent features and subtle details.
Authors
Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang
Abstract
Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: https://github.com/intelligolabs/RoSA