Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
2026-07-27 • Human-Computer Interaction
Human-Computer InteractionComputation and Language
AI summaryⓘ
The authors created FacialTalker, a system that makes computer speech sound more natural and expressive by understanding facial expressions during conversations. They developed AUTokenizer to turn each facial expression into a simple code for the system to understand. They also introduced a special training method called DualDPO to help the model learn both speech and facial cues better. To train and test their system, the authors built a large new dataset called VSDD-1K with thousands of hours of real people talking and showing facial expressions. Their experiments showed that FacialTalker improves how well computers recognize facial expressions and generate speech that fits the conversation.
Conversational Speech SynthesisFacial ExpressionsVisual TokenizerAction UnitsDual Direct Preference OptimizationMultimodal LearningLarge Language ModelsMultimodal Dialogue DatasetSpeech GenerationExpressive Speech
Authors
Yifan Hu, Shuwei He, Rui Liu, Haizhou Li
Abstract
Conversational Speech Synthesis is a fundamental component of human-computer interaction, aiming to generate contextually appropriate, expressive, and empathetic speech. However, facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction, whereas existing approaches often overlook this important modality. In addition, the lack of large-scale natural conversational datasets with both speech and visual modalities also limits the development of visual affect understanding in conversational settings.To address these limitations, we propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone. To efficiently encode facial expressions, we propose AUTokenizer, a single-codebook visual tokenizer that discretizes each frame-level facial expression into a compact token, trained with supervision from combinations of facial Action Units. We further introduce a dual direct preference optimization (DualDPO) strategy, which extends the DPO by jointly imposing preference constraints on both visual and speech token sequences, to enhance the model's understanding of facial expressions and speech semantics in multimodal conversational contexts. Moreover, we construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations, comprising over 1,033 hours of synchronized speaker videos and speech, with more than 85\% of frames containing valid faces. Extensive objective and subjective experiments demonstrate that FacialTalker consistently outperforms strong baselines in facial-expression perception and speech synthesis quality, generating speech that is more natural, expressive, and better aligned with the conversational context. The results also validate the effectiveness of our training strategy and dataset construction pipeline.