Speech emotion recognition improved by linking audio to editable concepts
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
SoundArtificial IntelligenceComputation and Language
Summary
Speech emotion recognition is about figuring out the feelings behind spoken words, but it’s still hard to do well. The authors study how large language models make these predictions using words, sound features, and speaker traits combined in clear, editable parts called concepts. They find that focusing too much on transcribed words can make the models worse at recognizing emotions. Their experiments show that when certain sound features are removed, the models’ individual mood guesses change, even if overall scores look similar. This means that just looking at overall accuracy misses important details about how the models think.
What this means in practice
- •For voice assistant developers: Improve emotion detection by making speech recognition models explainable through concept-based reasoning from both words and voice features.
- •For call center analytics teams: Use concept bottleneck models to identify how specific speech traits affect emotion predictions, enhancing customer sentiment analysis.
Authors
Hezhao Zhang, Thomas Hain
Abstract
Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.