Speech emotion labels differ when acted versus heard by listeners
Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente
SoundComputation and Language
Summary
Labels for emotions in speech usually show just one answer, but the feeling acted by the speaker and the feeling heard by listeners can be different. This paper looks at an Italian speech emotion dataset that has both the actor's intended emotion and what five different listeners thought. The researchers found that it's easier to identify the intended emotion than the listeners' most common choice, but each type of label tells a different story. Training models with full listener votes captures uncertain or mixed emotions better than just picking one label.
What this means in practice
- •For internet sound platform developers: Design emotion recognition tools that explicitly handle intended emotions and listener disagreements for user-facing sound applications.
- •For voice assistant builders: Improve emotional understanding in assistants by modeling listener uncertainty instead of relying on single emotion labels.
Tested on one dataset.
Authors
Marco Giordano, Stefano Giacomelli, Claudia Rinaldi, Fabio Graziosi
Abstract
Ground truth in Speech Emotion Recognition (SER) is often presented as a single categorical fact, yet an acted emotion and the emotion heard by listeners describe different aspects of the same recording. We examine this distinction in Emozionalmente, an acted Italian corpus that provides both an elicited production label and five listener judgements for every clip. Using frozen XLS-R, HuBERT, and WavLM representations with matched probes, we compare three label authorities: the expressed condition, the listeners' hard plurality, and their complete vote distribution. The results tell a consistent story. Recovering the expressed condition is easier than predicting a tie-broken listener plurality, but that gap compares two different questions rather than revealing a general loss of SER ability. As listener agreement weakens, the plurality becomes an increasingly brittle hard target. Training on the full vote distribution does not substantially change which class wins the model's hard decision, but it produces probability assignments that more closely follow the alternatives selected by listeners, especially for ambiguous clips. No authority therefore replaces the others. Expressed labels describe how speech was produced, pluralities provide an operational single-class decision, and vote distributions preserve perceptual uncertainty. For human-facing Internet of Sounds applications, evaluations should state which authority is being predicted, retain its aggregation rule and agreement information, and use distributional scores when disagreement is part of the target.