Dense retrievers struggle with mixed text and image document searches

Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Information Retrieval

Summary

Finding relevant information in mixed collections of text and images is tricky for current computer systems. The authors show that these systems prefer text and can get confused when both text and images are present. This confusion leads to poorer search results, especially when irrelevant text distracts the system more than irrelevant images. They propose a new method called Trident to balance the importance of text and images, improving search accuracy in mixed collections.

What this means in practice

  • For search engine developers: Improve search engines to handle collections containing text, images, and their combinations by reducing bias toward text and enhancing mixed-modality search quality.
  • For digital archive managers: Manage and retrieve documents with mixed content types more effectively by applying balanced retrieval methods that address modality mixing challenges.

Authors

Yubo Sun, Chunyi Peng, Yukun Yan, Zhenghao Liu, Zhipeng Xu, Sen Mei, Linlin Xin, Zheni Zeng, Maosong Sun

Abstract

Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term Chaos in the Text. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce Trident, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.