Multimodal sentiment analysis improves with better video processing and timing
Fine-Grained Visual Preprocessing and Dual-Stream Temporal Modeling for Multimodal Sentiment Analysis on Social Media
Computer Vision and Pattern Recognition
Summary
Understanding feelings from social media videos is hard because videos can be noisy and the timing of facial movements is tricky to interpret. The authors created a system that cleans up video faces, tracks lip movements, and uses smart methods to read both still images and motion over time. They combined this with text analysis to improve how well computers can guess emotions. Their approach worked better than just making the system more complicated when data is limited.
What this means in practice
- •For social media platform developers: Improve sentiment detection in user videos by applying refined face and motion analysis to reduce noise and better capture emotions.
- •For customer support teams: Enhance automated understanding of customer emotions from video messages by integrating improved visual preprocessing and text-video fusion techniques.
Authors
Su Li, Yigong Zhang, Lei Xiong, Chune Li
Abstract
Multimodal sentiment analysis often remains text-dominant due to raw-video noise and insufficient temporal modeling. Using CH-SIMS v2.0S, this study proposes three improvements: the NAPS pipeline---a seven-stage system integrating face tracking,identity embedding, and normalized lip-motion analysis to reduce visual noise;DS-TANet, combining an EfficientNetB2 static stream, RAFT optical-flow motion stream, motion-guided attention, and Bi-GRU temporal modeling; and DS-TAFNet, fusing visual and MacBERT-Base textual representations via concatenation fusion. With NAPS, the static visual baseline achieves 80.98\% Macro F1, comparable to the text baseline of 80.55\%; DS-TANet improves visual Macro F1 to 82.58\%;and DS-TAFNet achieves 87.49\% accuracy and 87.48\% Macro F1. These results demonstrate that improving visual input quality and temporal representation is more effective than increasing fusion complexity under limited-data conditions.