Summary
Understanding emotions from social media videos is often hard because video data can be noisy and the timing of facial movements is tricky to analyze. This study shows that carefully cleaning up video inputs, such as tracking faces and normalizing lip movements, helps reduce errors. The authors also improved how the model watches video sequences over time by combining static images and motion information. When they combined these better video methods with text analysis, their system understood emotions more accurately. Their work suggests focusing on better video quality and time-based modeling works better than just making the combination of text and video more complex.
Multimodal sentiment analysisVisual preprocessingTemporal modelingFace trackingLip-motion analysisOptical flowEfficientNetBi-GRUFusionMacro F1 score
Authors
Su Li, Yigong Zhang, Lei Xiong, Chune Li
Abstract
Multimodal sentiment analysis often remains text-dominant due to raw-video noise and insufficient temporal modeling. Using CH-SIMS v2.0S, this study proposes three improvements: the NAPS pipeline---a seven-stage system integrating face tracking,identity embedding, and normalized lip-motion analysis to reduce visual noise;DS-TANet, combining an EfficientNetB2 static stream, RAFT optical-flow motion stream, motion-guided attention, and Bi-GRU temporal modeling; and DS-TAFNet, fusing visual and MacBERT-Base textual representations via concatenation fusion. With NAPS, the static visual baseline achieves 80.98\% Macro F1, comparable to the text baseline of 80.55\%; DS-TANet improves visual Macro F1 to 82.58\%;and DS-TAFNet achieves 87.49\% accuracy and 87.48\% Macro F1. These results demonstrate that improving visual input quality and temporal representation is more effective than increasing fusion complexity under limited-data conditions.