Voice native AI models detect frustration but ignore its meaning
Voice-Native LLMs Hear Frustration but Do Not Listen to It
SoundArtificial Intelligence
Summary
People show frustration both in the words they say and how they say them. The authors studied AI models that listen directly to voice recordings instead of just reading transcripts. They found these models can sense when a voice sounds frustrated but don’t actually use that feeling to change their answers. So, the AI hears frustration but doesn’t truly understand or respond to it. Their work included a special dataset called VoxAffect with many voice clips showing different emotions.
What this means in practice
- •For customer support teams: Improve voice assistant designs by recognizing that current models detect frustration tones but do not adjust responses accordingly.
- •For voice assistant developers: Develop new feedback systems that better leverage vocal emotion cues beyond detecting anger to enhance user experience.
Authors
Hanjun Luo, Yuhan Lu, Hanan Salam
Abstract
Users convey frustration in what they say and in how they say it, and voice-native models, which hear speech instead of an ASR transcript, can register both. Existing evaluations read unchanged performance as robustness without checking perception. Following Brunswik's lens model, we measure on identical audio whether frustration is expressed, perceived, and utilized. We cross frustrated wording with frustrated voice in a within-item 2x2 design over 353 tasks and release the 1,412 clips as VoxAffect. Across 8 models, outputs look robust, with no detectable vocal effect on accuracy for 7. With neutral wording, all 8 label the frustrated voice frustrated in at most 3.1% of clips, although 7 of them detect acted anger in 73-100% of clips and human listeners classify the voice correctly in 65.3% of judgments. Yet linear probes decode the frustrated voice from the audio encoders of the 2 open-weight models, with paired accuracy 0.67 and 0.69, and in proprietary models too the voice raises perceived frustration once the words agree. The models thus hear the frustrated voice but do not listen to it, whereas their invariance to frustrated wording, perceived in 40-75% of clips, is genuine robustness. Our work is openly accessible at https://github.com/Astarojth/hear-not-listen.