Tts adapts speaking style dynamically for smoother conversations
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sound
Summary
Text-to-speech (TTS) systems often struggle to change their speaking style naturally during multi-turn conversations. The authors introduce Interactive TTS, which separates style decisions from the speech generation process and makes style changes more controllable and clear. They also develop new techniques to better follow style instructions and keep the voice consistent across multiple turns. Their experiments show this method produces more natural and context-appropriate speech compared to previous models.
What this means in practice
- •For virtual assistant developers: Generate expressive and consistent speech across multi-turn interactions improving user experience in conversational AI.$Commercial implications: Enables more natural and adaptive digital assistants marketed in consumer and enterprise voice products.
- •For animated character studios: Create dynamically styled speech for characters in interactive media that adapts naturally to dialogue context.
Authors
Wenjie Tian, Kangxiang Xia, Jingbin Hu, Xinfa Zhu, HangRui Hu, Ziyue Jiang, Kexin Huang, Ting He, Lei Xie, Jin Xu
Abstract
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/