Multimodal learning avoids information loss with decomposed representations

Asymmetric Multimodal Learning with Decomposed JEPAs

Machine LearningComputer Vision and Pattern Recognition

Summary

Current methods that try to make different types of data like images and text 'match up' lose important unique information when one type can't fully express what the other contains. The authors show that just aligning these data types misses key details and propose a new approach called D-JEPA. This method keeps shared information together but also preserves unique parts of each data type separately. Their approach helps AI systems understand complex tasks better, like detailed visual question answering, by preventing them from losing important details.

What this means in practice

  • For multimodal ai engineers: build AI models that better retain private and shared information across image and text inputs for improved complex reasoning tasks.
  • For machine learning platform teams: integrate decomposed latent representations to enhance multimodal data processing pipelines and avoid semantic information loss.
  • For multimedia content service providers: deliver richer visual question answering features by preserving detailed modality-specific content alongside aligned representations.$Commercial implications: Enables improved visual question answering products that capture nuanced information, appealing to customers needing precise multimedia analysis.

Authors

Yusuf Shihata

Abstract

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.