Summary
Many computer models can tell if a caption matches an image by turning each into a single summary vector, but they often miss if the words and image parts match properly and in the right order. The authors show that the original model already holds information about which words go with which image regions and their arrangement, and the problem is to unlock that. They created BindCLIP, a method that cleverly matches words to image patches, considering multiple image layers and word order, to better judge if the caption correctly describes the image. This improves performance on tests involving understanding relations between objects, without needing complex extra tools or training.
What this means in practice
- •For image analysis developers: Integrate BindCLIP to improve accuracy of matching images with detailed textual descriptions without extra labeled data or complex models.
- •For content moderation teams: Use BindCLIP to detect when image captions misrepresent visual content by checking detailed word-region correspondences and spatial relations.
Authors
Liuyang Song, Yi Zhang, Zhongyi Deng, Daqian Yang, Hongbo Zhang
Abstract
Global vision--language similarities compress an image and a caption into one vector, preserving semantics but not which word corresponds to which region or how those regions are arranged; a model can recognize every word and object yet prefer a compositionally incorrect caption. We argue that a frozen encoder retains this association structure, so the problem is to read it rather than to rebuild it beside the pretrained similarity. We introduce BindCLIP, a pairwise scorer built on one latent object: a balanced token--patch--depth optimal-transport coupling that places both candidate captions and several visual depths in a single plan. Semantic, entity, order, and spatial evidence are read as energies of this state, and exchanging the candidates permutes the plan, making the score exactly antisymmetric. A geometric refinement inside the coupling contracts moves that the candidates and the visual depths do not support. No task label, parser, relation inventory, or detector is used. One checkpoint and one inference path improve the official What'sUp, ARO, and SugarCrepe benchmarks over frozen global CLIP, with the strongest transfer on the relation splits. Controls rule out patch access and caption-length shortcuts, and an inference-time lesion localizes spatial arrangement to the coupling.