Summary
Getting computers to handle conversations with natural features like overlapping speech and interruptions requires special training data. This paper introduces a method called ConversationalVoice that starts with recordings of real two-person conversations and creates cleaned-up, speaker-separated audio, rebuilt dialogue that keeps timing and speaker style, and new conversation examples that fit the original context. The authors show that their approach keeps the voices clear and natural sounding, and the generated conversations remain coherent and similar in interaction style. They focus on producing good training data and leave training full-duplex speech models for future work.
What this means in practice
- •For voice assistant developers: Create large-scale, high-quality training data that captures realistic conversation timing and speaker overlaps for building more natural voice assistants.
- •For call center automation teams: Generate training examples reflecting real overlapping speech and interruptions to improve automated conversation understanding and agent assistance systems.
Authors
Richard Yucheng He, Baodong Cao, Chen Xu, Yihang Liu, Tairan Chen
Abstract
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordings. We present Conversational Voice, a pipeline that converts real two-speaker excerpts into three complementary training-data artifacts. (1) Separation recovers speaker-specific tracks with stable speaker assignments, a canonical transcript, and naturally observed interaction timing. (2) Reconstruction generates speech in matched voices from a fixed source transcript, reconstructs the source turn order, pauses, and overlaps, and adds word-level alignment and delivery instructions. (3) Expansion generates new dialogue constrained by the source context, speakers, and observed interaction pattern. Automatic speaker-verification metrics remain strong across stages, with same-speaker similarity of 0.983-0.991 and positive discrimination margins of 0.199-0.209. Predicted speech quality (NISQA MOS) is 3.56 for separation, 4.41 for reconstruction, and 4.61 for expansion. A Gemini-based automatic evaluator assigns expansion mean scores of 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness. Expansion and reconstruction exhibit broadly similar interaction profiles; expansion's turn, overlap-event, backchannel, and interruption rates are 4.6%, 8.0%, 13.2%, and 16.0% lower, respectively. We evaluate data properties only; downstream gains in full-duplex model training remain for future work.