Text guided experts improve video sentiment analysis efficiency

Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis

Computer Vision and Pattern Recognition

Summary

Understanding feelings in videos is tricky because people use words, sounds, and facial expressions together, but words usually tell us the main message. The paper introduces a new method that trusts words most and carefully uses sound and video clues only when they seem reliable, making the system smarter and simpler. The authors use a way to measure how sure each clue is and combine them in a way that gives better results with less computing power. Tests on popular datasets show this method beats others on most measures while being much faster and lighter.

What this means in practice

  • For multimedia app developers: Build video apps that understand user emotions more accurately with less computational cost by selectively trusting text and nonverbal cues.
  • For social media content analysts: Improve automatic sentiment detection in user videos by reliably integrating spoken language with visual and audio signals while reducing system complexity.

Authors

Xiaohua Huang, Bingrui Lei, Xiuyan hao, Ling Zhou, Yuan Zong, Wenming Zheng, Guoying Zhao

Abstract

Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities. Existing methods often treat modalities as symmetric, overlooking the fact that language provides a dominant semantic prior while nonverbal signals are complementary and noise-prone. To address this, we propose a Text-Anchored Modality Experts with Uncertainty-Guided Routing, termed TAME, that reformulates MSA as an uncertainty-guided refinement paradigm, where the language prior is calibrated and enhanced by selectively trusted nonverbal evidence. Specifically, TAME adopts an asymmetric Mixture-of-Modality-Experts framework, in which textual features act as a fixed and always-activated shared expert, while audio-textual and video-textual cross-modal branches serve as dynamically routed auxiliary experts. Furthermore, we introduce lightweight Prophet Heads to quantify the sample-level predictive uncertainty of each individual modality, and this uncertainty metric is further incorporated into the modality router to realize reliability-aware expert aggregation and fusion. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that TAME achieves state-of-the-art performance on the majority of metrics while reducing parameter count by up to 78.7% and FLOPs by up to 41.2% over the mixture-of-expert baseline. %These results confirm that asymmetric expert construction with uncertainty-guided routing is an effective paradigm for MSA.