Audio visual model improves emotion recognition with auto segmentation
AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning
MultimediaSound
Summary
This paper tackles the problem of recognizing emotions from both speech and facial expressions in videos without needing humans to mark where each spoken sentence starts and ends. The authors create a system called AGF-Net that automatically cuts continuous dialogues into segments and uses a clever mix of audio and visual clues to guess emotions. Their design helps balance and fuse data from sound and visuals even when one is noisy or less clear. Tests show the system performs well even with noisy inputs, making it more practical for real-world use.
What this means in practice
- •For voice assistant developers: Improve emotion detection in voice assistants that operate on continuous, unsegmented conversations without manual input on speech boundaries.
- •For security monitoring teams: Use the system to automatically analyze emotions from surveillance audio and video where utterance boundaries are unknown and noisy environments prevail.
Authors
Chuan Han Li, Yun Chien Cheng
Abstract
This study develops an audio-visual emotion recognition system that integrates automatic utterance segmentation with multimodal fusion. Unlike conventional methods that rely on manually annotated utterance boundaries, the proposed system automatically generates candidate segments from continuous dialogue and extracts the corresponding audio and visual features for emotion classification. To address the challenges introduced by automatic segmentation, the proposed AGF-Net integrates Audio-Guided Fusion (AGF), Multimodal Negative Learning (MNL), and Predictive Dynamic Fusion (PDF) to improve cross-modal fusion, modality balance, and robustness to varying modality quality. Experimental results show that AGF-Net achieves 64.92% Accuracy and 65.47% F1-score under clean conditions, 62.13% Accuracy and 63.08% F1-score under moderate noise, and 58.19% Accuracy and 59.41% F1-score under higher noise. Overall, the proposed framework reduces reliance on manually annotated utterance boundaries and maintains competitive performance under automatically segmented and noisy audio-visual inputs.