Multimodal learning improves by selectively treating shifted information
Beyond Invariance: Factor-wise Selective Treatment of Shifts for Multimodal Learning
Computation and LanguageMachine Learning
Summary
Handling changes in data from different environments is a big challenge for AI systems that learn from multiple types of information like videos and audio. Instead of treating all changing parts of the data the same way, the authors propose a method called FaShift that figures out which parts of the data to ignore, keep, or partially use. This approach helps AI models make better predictions even when tested on new, different situations. They tested FaShift on several datasets and showed it improves accuracy and reliability compared to other methods.
What this means in practice
- •For multimodal ai developers: Build AI models that better handle shifts in data from different environments by selectively adjusting parts of their learned features.
- •For speech and video analytics teams: Improve the robustness of sentiment and behavior analysis tools on multimodal data across varied real-world conditions.
Authors
Guimin Hu, Qiwei Peng
Abstract
Distribution shifts pose a fundamental challenge to robust multimodal learning. Existing methods typically promote invariance by suppressing environment-dependent variations, implicitly applying a uniform treatment to shifted information. However, shifted information can have different predictive consequences: some contributes little to prediction, some remains reliably informative, and some exhibits environment-dependent predictive effects. This raises a fundamental question: which shifted information should be suppressed, preserved, or attenuated? To address this question, we propose FaShift, a factor-wise shift framework that diagnoses latent representation factors according to their distributional changes, intervention effects, and cross-environment stability. By factorizing multimodal representations and evaluating these complementary properties, FaShift distinguishes non-predictive factors, stably predictive factors, and predictively useful but unstable factors. It then selectively suppresses, preserves, or attenuates their contributions, rather than enforcing uniform invariance. Experiments on MOSI, MOSEI, and CH-SIMS show that FaShift achieves the best ID accuracy and F1 across all three datasets. Compared with the strongest competitors reporting OOD results, it improves OOD accuracy/F1 by 1.5/3.6 points on MOSI and 1.7/2.7 points on MOSEI, while achieving the best reported OOD accuracy and F1 on CH-SIMS.