Captionfuse improves product search by filling missing image details
CaptionFuse: Training-Free Composed Product Retrieval Via VLM Intent Captioning and Weighted Embedding Fusion
Computer Vision and Pattern Recognition
Summary
Searching for products by combining a picture and a description is hard when people only specify changes but not all product details. The authors designed a way to generate detailed product captions by having a vision-language AI describe both the picture and the requested changes together. This helps the search system know what to keep from the image as well as what to change, improving accuracy without extra training. They showed captions with the right attributes work better than raw user text.
What this means in practice
- •For ecommerce platform developers: Improve product search accuracy by automatically generating detailed product descriptions from images and user modification text without retraining models.$Commercial implications: Enables better search features that can be integrated into retail platforms, enhancing customer experience and increasing sales through more relevant product retrieval.
- •For digital marketing teams: Create richer product captions from images and user intents to improve online catalog presentation and retrieval without additional training costs.
Authors
Zhao Gao, Mina Metias, Mohamed Noordeen Alaudeen, Bassel Saleh
Abstract
Composed Image Retrieval (CIR) retrieves products matching a reference image modified by natural language---a core capability for eCommerce search. Existing training-free methods fuse CLIP embeddings via arithmetic, but we identify a fundamental limitation: user text modifications are attribute-sparse---a user who types ``make it red with shorter sleeves'' specifies what to change but not what to keep (the floral pattern, the V-neck, the fit-and-flare silhouette). We introduce language-space fusion: a Vision-Language Model (VLM) jointly interprets the reference image and modification to generate a visually grounded product-title caption---e.g., ``Red floral fit-and-flare dress with short sleeves and V-neck''---that fills in the missing attributes from the image. Our method, CaptionFuse, requires no task-specific training and is model-agnostic. A controlled experiment confirms that attribute content--- not linguistic style---drives the gains: imperative and descriptive captions with identical attributes achieve equal R@10, while both outperform raw modifications by +6.5 points.