Listener facial reactions improve planning of conversational speech
Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionSound
Summary
Good conversations depend not just on words but also on how people listen and react. This paper shows a way to use a person’s facial expressions right before a response to help decide how the next spoken reply should sound and feel. The authors found that including these facial cues made generated speech feel more natural and better matched the conversation. They combined this with a system that turns planned speech style into audio, making it more lifelike. Their code is publicly available for others to build on.
What this means in practice
- •For virtual assistant developers: Create conversational agents that adjust emotional tone in replies based on user facial reactions for more natural interactions.
- •For videogame voice designers: Generate in-game character speech that adapts prosody and emotion from player facial cues to enhance immersion.
Authors
Yunji Chu
Abstract
Conversational speech depends on dialogue context and the listener's immediately preceding behavior. We propose ReACT-TTS, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization. On a strict dyadic MELD protocol, Temporal conditioning yields higher mean macro-F1 and VAD concordance than Text-only across ten seeds, while accuracy remains essentially unchanged. Ablations show that temporal modeling performs best among the visual variants and that an explicit early-to-late difference is unnecessary; correct listener reactions also outperform cyclic mismatches on average. In a contextual-appropriateness study with 20 speech researchers, 76% of judgments prefer Temporal, 9% Text-only, and 15% report no preference. We further connect the predicted response style to a Grad-TTS backbone for end-to-end speech realization. Overall, the results support pre-response listener dynamics as complementary cues for conversational response planning. The source code is available at https://github.com/CYJ1/ReACT-TTS_public.