Vision language models get calibrated scores for matching detailed text and images
Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models
Computer Vision and Pattern RecognitionComputation and Language
Summary
Vision-language models need to decide how well long, detailed texts match images, but their similarity scores can be hard to interpret. The authors introduce a simple method called Congruency Score (CS) that converts these raw scores into clearer, bounded measurements. They tested it on several datasets and models, showing that improving one part of the system doesn't always make the matching better. Their work shows that scoring text-image matches is about balancing retrieval accuracy, human judgment alignment, and consistent score thresholds.
What this means in practice
- •For computer vision engineers: Provide calibrated congruence scores to improve evaluation and filtering of long textual descriptions against images in vision-language pipelines.
- •For natural language processing developers: Incorporate calibrated image-text congruence scoring to enhance multimodal retrieval and alignment tasks involving long text inputs.
Authors
Alessandro Gambetti, Qiwei Han
Abstract
Long-text image--text congruence scoring is increasingly important for vision-language systems that must evaluate whether detailed textual descriptions match visual content. However, raw similarity scores from dual-encoder models are difficult to interpret as calibrated congruence measures, especially under the modality gap between image and text embeddings. This paper proposes Congruency Score (CS), a lightweight calibration layer that maps image--text similarity evidence into a bounded score. Using DOCCI and Urban1k, we evaluate four frozen vision-language backbones and show that observed reductions in post-projection centroid distance do not uniformly improve image--text retrieval performance. Human-grounded evaluations on DOCCI further reveal a trade-off: direct post-hoc calibration preserves high association with human judgments, whereas selected projection-based configurations can reduce threshold-relevant slope and intercept distortions at the cost of retrieval performance and association strength. These results establish long-text image--text congruence scoring as a calibrated score-estimation problem, where retrieval performance, human association, and threshold calibration must be evaluated as distinct objectives. CS provides a lightweight way to expose and operationalize this separation.