Training-Free Pseudo-Fusion for Composed Image Retrieval with Diffusion Models and Multimodal Large Language Models
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionInformation Retrieval
AI summaryⓘ
The authors introduce PeFuse, a new way to search for images based on both a picture and some text without needing extra training. Instead of combining the picture and text into one fancy model, they change the problem so it only needs searching using one type of input at a time. They use existing AI tools that generate images and understand language to do this. Their tests show this method works well compared to other approaches and is flexible because its parts can be swapped out easily.
Composed Image RetrievalContent-based Image RetrievalMultimodal FusionDiffusion ModelsLarge Language ModelsZero-shot LearningText-to-Image RetrievalSingle-Modality RetrievalCross-Modal RetrievalGenerative Conversion
Authors
Fan Xu, Luis A. Leiva
Abstract
Composed Image Retrieval (CIR) is an emerging paradigm in content-based image retrieval that enables users to formulate compositional queries by combining a reference image with an auxiliary modality, usually text-based. This approach supports fine-grained search where the target image shares structural elements with the user-provided image while incorporating the modifications specified by the auxiliary text. Conventional CIR methods rely on multimodal fusion to combine visual and textual features into a joint query embedding, which requires training modules that align composed queries with the targets. In this work, we propose PeFuse (for pseudo-fusion), a training-free framework that leverages pretrained Diffusion Models and Multimodal Large Language Models to bridge modalities via generative conversion. We introduce two novel strategies: uni-directional and bi-directional conversion, which convert CIR into four single-modality retrieval problems. These methods reformulate CIR as either intra-modal or cross-modal single-query retrieval tasks, bypassing the need for dedicated task-specific training. Extensive experiments on standard benchmarks demonstrate that converting CIR into text-to-image retrieval tasks is more effective than alternative conversion strategies, achieving competitive or superior performance compared with state-of-the-art methods, while maintaining high flexibility thanks to replaceable components of the conversion pipeline. These results highlight the effectiveness of the pseudo-fusion paradigm for zero-shot CIR. Our code is publicly available at: https://github.com/StevenXuf/PeFuse4CIR.