Lightweight speech and text model improves real-time emotion detection
RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems
Artificial IntelligenceMachine LearningMultimediaSound
Summary
Detecting emotions from speech and text usually requires complex and slow computer models. The authors created a lighter and faster system that mixes speech and text information in a simpler way, making it easier to use in real-time monitoring, like in surveillance. Their method uses a one-way attention approach to add emotional speech clues to text, which reduces computing needs but still keeps good accuracy. Tests showed it works better than some common methods while running faster and using fewer resources.
What this means in practice
- •For surveillance system engineers: Integrate a lightweight emotion recognition module that processes speech and text efficiently for behavioral monitoring in real-time surveillance setups.
- •For embedded system developers: Build emotion detection features into resource-constrained devices by using the model’s low-parameter, fast inference design for real-time applications.
Authors
Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen, Phuc-Lu Le
Abstract
Recent multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight multimodal SER framework featuring an asymmetric Residual Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective speech cues into semantic text representations through a one-directional residual attention pathway. Combined with a BYOL-inspired cross-modal alignment objective and attention-guided pooling, the proposed framework improves multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric multimodal fusion is an effective alternative to interaction-heavy cross-modal transformers for real-time surveillance applications.