Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied contrastive learning when working with three or more types of data (modalities) instead of just two, like images and text. They found that how the encoders behave mathematically, specifically their Jacobian condition numbers, affects how well the model learns to align different data types. By improving this mathematical property using special techniques like Jacobian regularization and tweaks to the network design, they made the models work better on tasks involving multiple data sources. Their work shows that good learning depends on both the training goals and how the encoders are built and optimized.
contrastive learningmultimodal learningencoder Jacobianconditioningsingular valuesLeakyReLUresidual connectionsregularizationlinear probemultimodal alignment
Authors
Tillmann Rheude, Roland Eils, Benjamin Wild
Abstract
Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimodal alignment can introduce optimization and representation challenges. We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning: poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal alignment. We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization and demonstrating that simple modifications like LeakyReLU activations and residual paths recover these geometric benefits. Across a synthetic benchmark and four real-world datasets including missing modalities, improving Jacobian conditioning boosts retrieval and linear probe performance across multiple contrastive objectives, whereas expressive objectives yield little benefit in linear probes. More broadly, our results show that multimodal contrastive learning depends not only on objective expressivity, but also on the geometric and optimization properties of the underlying encoders.