Flow-CLIP improves image and text matching with refined visual clues
Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching
Computer Vision and Pattern Recognition
Summary
Matching pictures with sentences is hard when sentences describe similar objects in different ways. The authors introduce Flow-CLIP, a method that looks closely at details in images to better understand how words relate to specific parts of a picture. They adjust the overall match score by seeing how well each word pairs with image patches. Their approach also trains the model to distinguish correct captions from wrong ones more reliably by comparing across multiple images. This makes matching more accurate, especially when objects and their descriptions mix in complex ways.
What this means in practice
- •For multimedia search developers: Enhance image search engines by improving the accuracy of matching complex captions to images using detailed local visual context.
- •For automated content tagging teams: Improve automated tagging of images with precise attribute-object combinations by refining the connection between text descriptions and image parts.
Authors
Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
Abstract
Vision--language pre-training has enabled substantial progress in image--text retrieval and zero-shot recognition by learning transferable cross-modal representations. However, sentence-level alignment remains insufficient for compositional matching, where captions can describe the same objects and attributes while differing in how these elements are bound. To address this limitation, we propose Flow-CLIP, which refines global image-text alignment through image-conditioned changes in token--patch evidence. Global Compatibility Calibration (GCC) learns a sentence-level similarity metric, while Image-Conditioned Evidence Refinement (ICER) incorporates visual patch context into token queries and uses the resulting change in local matching evidence to adjust the global score. We further introduce Cross-Image Margin Ranking (CIMR), which compares the same positive--negative caption pair across matched and mismatched images, encouraging a larger ranking margin when the image supports the correct description. Experiments on compositional matching and attribute--object binding benchmarks show that Flow-CLIP improves binding accuracy over pretrained CLIP and learned global alignment through local evidence refinement, while cross-image ranking supervision mitigates reliance on language priors. Code will be released at link.