Shared sentiment evidence improves continuous multimodal sentiment prediction
Shared Sentiment Evidence from Frozen Multimodal Models for Continuous Sentiment Prediction
Multimedia
Summary
Predicting emotions from speech, text, and facial cues is tricky because these signals can disagree. The authors propose a method called Shared Sentiment Evidence (SSE) that uses fixed output representations from separate models to better combine different views of sentiment. Their approach improves continuous sentiment prediction on popular datasets without retraining the core models. This suggests it’s possible to learn sentiment continuously using reusable, shared outputs across various pretrained models.
What this means in practice
- •For voice assistant developers: Improve emotion recognition by integrating fixed shared sentiment representations from multiple pretrained models for smoother user interaction.
- •For customer service analytics teams: Use continuous multimodal sentiment predictions derived from shared model outputs to better assess caller emotions during support calls.
Authors
Shuobo Wang, Ruiqi Chen, Shaobo Qiao, Shixu Yang, Feng Xu
Abstract
Continuous multimodal sentiment analysis estimates polarity and intensity from linguistic, acoustic, and visual cues that may disagree. Hard judgments compress sentiment alternatives, while learned fusion interfaces tie task adaptation to model representations. We introduce Shared Sentiment Evidence (SSE), an output-space interface with fixed view and sentiment-option semantics across frozen backbones. Four queries yield five-option distributions whose probabilities, entropy, and cross-view relations form a 28-dimensional vector for CatBoost regression. On CH-SIMS-v2 and CMU-MOSI, the Qwen-based interface improves on supervised joint-only readout. Matched Ridge and CatBoost controls assess the contributions of additional views, deterministic summaries, and nonlinear readout. With the same interface specification and fixed readout settings, separately fitted heads also improve on joint-only prediction for three additional frozen backbones. These results support learning continuous sentiment from reusable output coordinates.