Multimodal learning method captures unique and shared data features
Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning
Computer Vision and Pattern RecognitionMachine Learning
Summary
Multimodal self-supervised learning faces the challenge of understanding different types of data like images, text, and audio together. The authors propose a new method called MIIB that separates what information is shared across data types, what is unique to each type, and what emerges only when combining them. This new approach helps create better data representations by focusing only on useful information and ignoring noise. They tested their method on various tasks and found it improved performance compared to earlier techniques.
What this means in practice
- •For multimedia app developers: Design apps that better combine images, text, and audio to improve user experience by capturing richer cross-modal information.
- •For healthcare data engineers: Integrate diverse medical data types more effectively to improve patient data representations and predictive models.
Authors
Liangjian Wen, Jiangtao Zheng, Jiang Duan, Qun Dai, Yong Dai, Jianzhuang Liu, Jun Wang, Shu-Dong Huang, Xinwang Liu, Zenglin Xu, Zhao Kang
Abstract
Multimodal self-supervised learning is challenged by heterogeneity, unequal informativeness, and complex information interactions across multimodal data sources such as images, text, audio, and tabular data. While contrastive methods effectively align shared semantics, they typically rely on strong multi-view redundancy assumptions, overlooking modality-specific and synergistic information, thus leading to suboptimal representations. We propose the Multimodal Information Interactions Bottleneck (MIIB), a principled framework grounded in Partial Information Decomposition that explicitly encodes redundant, unique, and synergistic information within multimodal interactions. MIIB employs a dual-path model architecture for modality-specific encoding and multimodal fusion, optimized using surrogate tractable entropy-based objectives that enforce minimal sufficient representations without requiring explicit mutual information estimation. By leveraging variational bounds and score-based entropy estimation, MIIB efficiently captures complex cross-modal interactions while suppressing superfluous information. Experiments across diverse benchmarks demonstrate consistent improvements over state-of-the-art multimodal self-supervised methods. Code is available at https://github.com/wanglingzaishi/MIIB.