Controllable multi-speaker dialogue speech improves with iterative refinement
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Computation and LanguageSound
Summary
Producing realistic conversations with different speakers in text-to-speech (TTS) systems is challenging because it needs natural voices, consistent speaker sound, smooth turn-taking, and control over emotions and speech styles. The authors developed a method where the system listens to its own output and makes improvements step-by-step instead of generating everything once. Their approach can fix mistakes in tone, speed, or loudness without redoing the whole conversation, leading to more natural and coherent multi-speaker dialogues. Tests on Chinese-English dialogues showed better quality and control compared to other methods.
What this means in practice
- •For voice assistant developers: Improve multi-speaker dialogue voices with precise control over emotion and timing for more natural conversations.
- •For game audio designers: Enable more consistent and controllable character speech in interactive stories with bilingual and multi-speaker dialogue.
Authors
Kangxiang Xia, Xinfa Zhu, HangRui Hu, Kexin Huang, Wenjie Tian, Ziyue Jiang, Bingshen Mu, Jingbin Hu, Ting He, Lei Xie, Jin Xu
Abstract
Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.