Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors focus on improving how computers understand emotions from different types of inputs like text, sound, and images. They point out that audio and visual signals show feelings over different time lengths, which many current methods miss by simplifying too much. Their new method, MGSI, looks at these signals over short, medium, and long times and uses text to better align and enhance the emotional information. Tested on several datasets, their approach outperforms similar models that rely on frozen language models and competes well with other advanced methods.
multimodal sentiment analysislarge language modelsaudio-visual signalstemporal scalescross-modal alignmentsentiment polaritysentiment intensitytext-guided refinementpseudo-tokensaffective computing
Authors
Shanshan Lin, Yuesheng Wu, Chao Chen, Yizhe Yang, Zhihao Chen, Zexian Yang, Xiangwen Liao
Abstract
Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.