Vision language models measured for caption quality and alignment gaps
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
Computer Vision and Pattern Recognition
Summary
When computers match pictures with words, they use a shared space to compare them, but pictures and words often don’t line up perfectly. The authors created UOT-Gap, a way to measure how much this mismatch happens without retraining the models. They found that as captions get worse, UOT-Gap clearly shows the drop in matching quality. This method helps spot how well image and text parts align and how robust the matching is to poor captions.
What this means in practice
- •For machine learning engineers: Identify and quantify the gap between image and text spaces to improve vision-language model robustness without retraining.
- •For data quality analysts: Detect degradation in caption quality automatically by measuring deviations from optimal transport-based alignment in vision-language embeddings.
Authors
Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie
Abstract
Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing accounts connect this modality gap to initialization, contrastive dynamics, and information imbalance, while its distributional and pairwise contributions to retrieval remain unresolved. We introduce UOT-Gap, a training-free variational diagnostic that models frozen image and text embeddings with unbalanced entropic optimal transport (UOT). The UOT optimum separates transport, coupling complexity, and marginal mass variation; a complementary pair-aware residual compares observed image-caption pairs with the UOT soft matching. On Flickr8K and COCO-1K with frozen CLIP, OpenCLIP, and SigLIP encoders, caption degradation reduces Flickr8K Recall@1 from 0.559 to 0.003. Across six dataset-model conditions, the pair-aware residual tracks retrieval degradation with mean absolute Spearman 0.973, compared with 0.392 for the mean gap. The association remains stable across five random COCO-1K subsets at $0.954\pm0.026$, with a minimum of 0.943. UOT barycentric updates reduce the transport objective while degrading retrieval, distinguishing geometric objective descent from task improvement. These results establish UOT-Gap as a diagnostic for caption quality, modality alignment, and retrieval robustness.