Lightweight multimodal system improves real-time emotion detection in surveillance
RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems
Artificial IntelligenceMachine LearningMultimediaSound
Summary
Detecting emotions from speech and text helps surveillance systems understand human behavior, but current methods are often too slow or require too much computing power. The authors introduce RAFM_SER++, a simpler way to combine speech and text information that uses less computing power while still being accurate. Their method only sends speech cues in one direction into text, avoiding complex back-and-forth processing. Tests show RAFM_SER++ is faster, smaller, and performs better than some existing systems, making it suitable for real-time monitoring.
Speech Emotion RecognitionMultimodal learningResidual Attention FusionCross-modal transformersBYOL (Bootstrap Your Own Latent)Attention-guided poolingIEMOCAP datasetESD datasetBalanced accuracy (BACC)Inference speed
Authors
Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen, Phuc-Lu Le
Abstract
Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.