Human voices boost emotion recognition over ai synthetic speech
A barrier or a booster? Familiarity effects on Mandarin emotion prosody recognition using AI-powered voice cloning
Human-Computer InteractionSound
Summary
Recognizing emotions from speech depends on how familiar we are with the speaker and whether the voice is natural or computer-generated. The authors found that people understand emotions more accurately and faster when listening to real human voices compared to AI-generated voices. However, having a familiar speaker didn’t affect heart rate measures, suggesting that recognizing emotions in synthetic speech is more mentally challenging. These findings show that current AI voices still struggle to convey emotions as clearly as human voices.
What this means in practice
- •For voice assistant developers: Improve emotion recognition features by prioritizing natural human voice data over synthetic voices for better user understanding.
- •For call center managers: Refine training systems by understanding limitations of AI-generated voices in conveying emotions to improve customer interactions.
Authors
Feng Xu, Gaoyuan Zhang, Shanshan Xue, Yixiang Chen, Hanrui Zhou, Xurong Xie, Hui Chen
Abstract
Emotion prosody perception requires simultaneous processing of acoustic cues and speaker identity. While listeners effortlessly decode natural speech, AI synthetic voices introduce cognitive complexities due to subtle acoustic atypicalities. It remains unclear how these synthetic features interact with a listener's prior social knowledge and memory of a familiar speaker. This study investigated how speech sources (human vs. AI) and speaker familiarity affect emotion recognition accuracy and cognitive load. A within-subject task with Mandarin-speaking adults evaluated behavioral (accuracy, reaction time) and physiological data (heart rate variability). Results showed that human voices yielded significantly higher accuracy and faster processing times than AI voices, while HRV did not significantly differentiate between conditions. These findings show that decoding synthetic speech is gated by top-down social cognition, highlighting limitations in current AI synthesis technologies.