Context-aware text to speech models infer emotion from conversation
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
Computation and LanguageSound
Summary
Generating speech that sounds natural means capturing the speaker's mood and style from the way people talk in conversation. This paper introduces a method where the system listens to past conversation audio and reasons about the emotional style before making speech from text. The researchers built a large bilingual dataset and created models that can produce speech matching emotions like stress or rhythm based on context. Their approach uses fewer parameters yet achieves good quality in emotion and style consistency compared to other big models.
What this means in practice
- •For voice assistant developers: Generate more natural and emotionally consistent speech in conversational AI systems by inferring speaking style from previous dialogue audio.$Commercial implications: Enables voice assistant products to sound more engaging and context-aware by automatically adapting speech emotion and style.
- •For audiobook producers: Create expressive synthetic narration that automatically adjusts emotional tone and speech rhythm based on the story context without manual annotation.
Authors
Weizhen Bian, Sitong Cheng, Rongxiu Zhong, Jiahao Pan, Liumeng Xue, Boyi Kang, Shilei Zhang, Jinglei Liu, Yue Wang, Junlan Feng, Bei Liu, Wei Xue
Abstract
Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated speech typically relies on clear user-specified instructions. In natural conversations, speaking style should be naturally inferred from the preceding conversational context. Therefore, we propose COT-TTS, a context-aware, reasoning-based text-to-speech task. Given historical conversation audio, target text, and a reference speech, the system should comprehend the conversational context, infer an explicit intermediate reasoning, and finally synthesize the target speech with the specified timbre. To support this task, we constructed a large-scale bilingual conversational speech dataset comprising 9 million training samples, including a high-quality subset of 1 million samples. We further constructed a source-disjoint benchmark with 800 human-verified samples and established strong task-specific baselines. Additionally, we developed end-to-end autoregressive models with parameter sizes of 0.6B and 1.7B, generating emotion-labeled transcripts, editable speech style inferences, and speech tokens. Experimental results show that the proposed model achieves performance comparable to large-scale baseline systems with significantly fewer parameters. At the same time, the model performs well in terms of duration consistency and emotional consistency, and can generate appropriate emotional, stress, and rhythmic variations based on the conversational context. To facilitate future research, we will publicly release the data construction pipeline, dataset, trained models, and related resources. The demo page and additional resources are available at https://luckybian.github.io/COT-TTS