Optimization causes uneven confidence in multimodal learning models
Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification
Machine Learning
Summary
Multimodal learning combines different types of data like images and text to make better predictions. The authors found that during training, some types of data become more confident than others, which leads to uneven contributions and worse overall results. They traced this problem to how the individual data types behave rather than the combined system itself. To fix this, they developed a method called Max Confidence Regularization that balances confidence across data types, improving accuracy. Their tests on common datasets showed this method worked better than existing approaches.
What this means in practice
- •For machine learning engineers: Improve multimodal classification systems by balancing confidence across data types to boost overall prediction accuracy.
- •For multimedia platform developers: Enhance content analysis tools by applying confidence regularization to better integrate video, audio, and text features.
Authors
Longfei Huang, Xiangyu Wu, Yang Yang
Abstract
Multimodal learning (MML) falls into the optimization dilemma due to the modality imbalance phenomenon, leading to suboptimal overall performance in practice. While many attempts primarily focus on balancing the optimization dynamics across modalities to address this issue, we identify a subtle yet critical flaw: optimization yields asymmetric gains in predictive certainty, with the strong modality more confident than the weak one, driving imbalanced modality contributions. In this paper, our analysis reveals that this flaw stems from unimodal characteristics rather than multimodal learning, and this confidence discrepancy can be corrected by positive cross-modal intervention. Based on this insight, we propose multimodal Max Confidence Regularization (MaxCR) to dynamically intervene in modality semantic confidence. Specifically, the semantic confidence of each modality is tracked using a nonlinear sparsity measure. We then design max suppression and max excitation based on this measure to regularize strong and weak modalities, respectively. They penalize and encourage the top-1 confidence, thereby constraining multimodal prediction. To this end, strong and weak modalities are expected to make calibrated confidence, thereby improving the overall performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines.