Audio models locate emotions precisely during speech segments
Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding
SoundArtificial Intelligence
Summary
Audio emotion recognition usually labels the entire recording with one emotion, which can be confusing when multiple people speak or emotions change over time. The authors changed this by linking emotions to exact parts of the speech and describing the tone used. They created new data and a method called M-TAG that helps models better understand when emotions happen during a conversation. Their tests show common models can guess emotions okay but fail to tell when they occur in the audio.
What this means in practice
- •For voice assistant developers: Improve virtual assistants by pinpointing exactly when emotions occur in conversations with multiple speakers or overlapping speech.
- •For call center technology teams: Enhance customer support by detecting emotional changes within calls to better understand and respond to caller feelings over time.
Authors
Abdelrahman Mohamed, Lars Kai Hansen, Zheng-Hua Tan
Abstract
Audio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions. To support this formulation, we curate temporally annotated versions of existing emotion recognition datasets and construct recordings containing two to four affective speech spans, including overlapping speech. Training in this longer format with a standard language-modeling objective can degrade both emotion recognition and temporal grounding performance, while tone descriptions can provide shortcuts for emotion prediction. To address these challenges, we introduce Masked Temporal Affective Grounding (M-TAG), a supervised training objective that combines full-sequence language modeling with emotion and timestamp cross-entropy losses under attention masking. The masking varies the context visible to emotion-label tokens to reduce reliance on shortcuts and improve generalization, while the timestamp loss incorporates a distance-aware weight to penalize larger temporal errors. We evaluate EMO-TAG, a model fine-tuned using our dataset and objective, on emotion recognition and affective temporal-grounding against three AER and audio-language baselines: Flamingo-Next, Audio-Reasoner, and AffectGPT. Our results show that existing models achieve limited affective temporal-grounding despite competitive emotion recognition performance.