Multimodal sentiment analysis improves reliability with noise and missing data
Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
Machine Learning
Summary
Understanding emotions from text, speech, and images is tough when some information is missing or noisy. The authors propose a method that can tell which parts of the data are unreliable and reduce their impact. They also use similar, high-quality examples to fill in gaps and improve emotion recognition. Their approach combines information from all sources more effectively, leading to better results even with imperfect data.
What this means in practice
- •For customer service teams: Improve automated systems that detect customer sentiment from calls and chats despite poor audio or missing visual cues.
- •For social media platforms: Enhance emotion detection in posts and videos even when some modalities like audio are missing or corrupted.
Authors
Menghua Jiang, Haokai Gao, Xiangui Kang, Haifeng Hu, Sijie Mai
Abstract
Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.