On the modality gap and the contrastive loss in multi-modal representation learning
2026-07-12 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study why image and text embeddings in CLIP-style models don't perfectly align, calling this the 'modality gap.' They identify that the common training method, InfoNCE, causes this gap especially at low temperatures when using separate encoders. They propose a new method, xNCE, that includes extra contrastive pairs and helps reduce this gap without hurting performance. Their method improves zero-shot classification compared to previous approaches while keeping retrieval quality high.
modality gapCLIPdual-encodercontrastive learningInfoNCEtemperaturexNCEzero-shot classificationretrieval performance
Authors
Fabian Mager, Hiba Nassar, Lars Kai Hansen
Abstract
We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.