Multimodal model improves balance and synergy for better predictions
MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information
Artificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Multimodal learning uses different types of data together, like images and text, but sometimes one type dominates and the model misses useful combined clues. The authors introduce a method called MSCR that balances how each data type contributes and encourages the model to find helpful interactions between them. This approach helps the model make better predictions, especially when single data types alone are confusing. They show their method works better on several standard tests and with modern fusion techniques.
What this means in practice
- •For multimodal ai engineers: Build models that better balance and combine multiple data types for improved accuracy on challenging samples.
- •For computer vision developers: Enhance vision-language systems by better exploiting interactions between image and text features during fusion.
Authors
Xinyu Chen, Liangjian Wen, Jiang Duan, Dongkai Wang, Yong Dai, Jiayu Bai, Jianzhuang Liu, Zhen Tian, Guoping Qiu, Zhao Kan
Abstract
In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions. Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training. These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion. Synergy is a major source of multimodal advantage, because it is predictive gain that cannot be recovered from unimodal evidence alone. We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information. MSCR has two complementary objectives. An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference. An adaptive gate further concentrates the regularization on samples that need stronger collaborative modeling. Experiments on five benchmarks show that MSCR consistently improves performance over MCR and strengthens synergistic capability. It raises conditional recovery, the accuracy of the fused prediction on samples that the unimodal predictors misclassify, as well as the share of the model's output that InterSHAP attributes to cross-modal interactions. Its gain in conditional recovery persists when the fusion module is Transformer-based. Code is available at https://github.com/chenxinyu991-collab/MSCR.