Speech models overcome text bias to better read vocal emotions
ACR-Net: Mitigating Semantic Dominance via Contrastive Acoustic-Semantic Decoupling
SoundArtificial Intelligence
Summary
Speech emotion recognition systems often get confused when the tone of voice says one thing but the words mean another. The authors created a new test set to challenge these models with conflicting tone and text. They also built a new method called ACR-Net that separates voice signals and word meanings into different spaces to avoid the models focusing too much on text. This lets the system correctly understand emotions even when tone and words disagree.
What this means in practice
- •For voice assistant developers: Improve voice assistants to better interpret user emotions when tone conflicts with spoken words.
- •For call center software teams: Enhance emotion detection in customer calls by reducing bias toward spoken text over vocal tone.
Authors
Mengke Zhang, Yanda Shao, Tianhe Wu, Kai Feng
Abstract
Speech Emotion Recognition (SER) models and Audio LLMs fail when vocal tone contradicts textual semantics. Extensive training on congruent data engenders a rigid semantic prior, causing "Semantic Dominance", a severe text bias during cross-modal conflicts. We introduce the ASPIRE benchmark of adversarial audio-text pairs across four conflict types, alongside two metrics: Semantic Overconfidence Penalty (SOP) and Latent Decoupling Degree (LDD). Unlike fusion methods that merely re-weight collapsed features, our Acoustic Conflict Resolution Network (ACR-Net) uses Cross-Modal Attention and Contrastive Decoupling Loss to disentangle contradictory representations into orthogonal latent spaces. Experiments show existing models fail under polarity conflicts due to extreme semantic overconfidence. Conversely, ACR-Net preserves acoustic fidelity, maintaining high LDD to drive SOP near zero, achieving superior Acoustic Accuracy (ACC) in both adversarial and standard scenarios.