Adaptive routing improves multimodal input understanding with fewer parameters
InstMoE: Adaptive Multimodal Routing with Specialized Experts
Computation and Language
Summary
Multimodal inputs, like videos with sound and text, need different ways of understanding. The authors developed InstMoE, a system that decides which expert parts should handle each input based on its details. To avoid confusion from irrelevant information, they added a module that aligns meanings and reduces distractions. This method performed better on tests for understanding sentiment in videos while using fewer resources. It also showed that each input prefers a unique way of being processed rather than using one fixed approach.
What this means in practice
- •For multimodal ai developers: Design systems that dynamically route inputs to specialized experts to improve understanding of audio, text, and video data with fewer parameters.
- •For sentiment analysis teams: Enhance sentiment prediction from multimodal data sources by applying adaptive routing to better capture task-relevant features.
Authors
Guimin Hu, Xiang He, Yingjian Li, Zheng Lian, Boyan Xu, Ruichu Cai
Abstract
Multimodal inputs are inherently heterogeneous, not only across modalities but also in the information pathways required for effective prediction. To address this limitation, we propose InstMoE, an adaptive expert routing framework for multimodal learning. InstMoE dynamically routes each input to specialized unimodal and cross-modal experts, allowing the model to adapt its information pathways to the characteristics of the input. However, routing can be misled when modality-specific variations obscure task-relevant semantics. Such irrelevant variations may distort routing decisions, causing inputs to be assigned to inappropriate experts. We therefore introduce a Contrastive Semantic Alignment module, which encourages semantically similar inputs to share task-relevant representations while suppressing irrelevant modality-specific variations. Experiments on multimodal sentiment analysis benchmarks demonstrate that InstMoE achieves state-of-the-art performance on CMU-MOSEI and CH-SIMS v2 while using substantially fewer parameters than competitive baselines. Further analysis shows that different inputs exhibit distinct expert preferences, demonstrating that InstMoE moves beyond fixed fusion toward adaptive multimodal computation.