Multimodal sentiment analysis method reveals hidden interaction value
I$^2$-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
Computer Vision and Pattern Recognition
Summary
Understanding feelings from words, tone, and facial expressions together is tricky because it’s hard to tell if the emotion comes from one part alone or their combination. The authors developed a new method called I²-KAN that keeps the information from each part and their interactions separate, making it easier to see what really matters. Their tests found that even parts that seem weak on their own can be very useful when combined with others. This helps better understand how different types of data work together to detect sentiment.
What this means in practice
- •For sentiment analysis engineers: Build systems that can separately analyze and explain the contribution of spoken words, tone, and facial cues to improve sentiment detection accuracy.
- •For customer experience teams: Implement advanced sentiment tools that reveal subtle emotional signals by capturing interactions across voice, text, and video in customer feedback.
Authors
Xiaoting Xie, Feng Qiao, Xianwei Zhang, Hailong Chen, Kaiwen Zheng, Joemon M. Jose, Xuri Ge, Yujun Li
Abstract
Multimodal sentiment analysis (MSA) aims to infer sentiment by integrating linguistic, acoustic, and visual information. However, many fusion methods integrate modality-specific representations and cross-modal interactions into joint representations, making it difficult to determine whether predictive utility arises from a modality itself or from its interactions with others. More fundamentally, a modality's standalone predictive utility may not faithfully reflect its interaction-conditioned predictive utility. To address this issue, we propose I$^2$-KAN, an interpretable multi-pathway framework that explicitly maintains intra-modal and pairwise inter-modal information in separate structural pathways. Each pathway is refined by a Task-Decoupled Linear-Gated Denoising Bottleneck (TDL-GDB) to learn compact and task-relevant representations, followed by structured Kolmogorov-Arnold Network fusion that retains explicit pathway associations and supports multi-level interpretation. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 demonstrate strong overall performance. Ablation and interpretability analyses consistently show across the three benchmarks that low standalone predictive utility does not necessarily imply low interaction-conditioned predictive utility: a modality that appears weak in isolation can remain informative through pairwise interactions. These findings highlight the interaction-dependent nature of multimodal utility and suggest that modality utility should be characterized from both standalone and interaction-conditioned perspectives.