Summary
Training computers to understand natural conversations with overlapping speech and interruptions is tricky because real recordings mix voices together. The authors developed ConversationalVoice, a method that separates each speaker's voice, recreates conversations with accurate timing and transcript details, and even generates new dialogue that sounds natural. Their method keeps the voices recognizable and the conversations realistic according to several quality checks. While the study focuses on creating the data itself, using this improved data to train full-duplex speech models is planned for future research.
full-duplex speech modelsturn-takingoverlapping speechspeaker separationspeech reconstructiondialogue generationspeaker verificationspeech quality metricsbackchannelsspeech alignment
Authors
Richard Yucheng He, Baodong Cao, Chen Xu, Yihang Liu, Tairan Chen
Abstract
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordings. We present Conversational Voice, a pipeline that converts real two-speaker excerpts into three complementary training-data artifacts. (1) Separation recovers speaker-specific tracks with stable speaker assignments, a canonical transcript, and naturally observed interaction timing. (2) Reconstruction generates speech in matched voices from a fixed source transcript, reconstructs the source turn order, pauses, and overlaps, and adds word-level alignment and delivery instructions. (3) Expansion generates new dialogue constrained by the source context, speakers, and observed interaction pattern. Automatic speaker-verification metrics remain strong across stages, with same-speaker similarity of 0.983-0.991 and positive discrimination margins of 0.199-0.209. Predicted speech quality (NISQA MOS) is 3.56 for separation, 4.41 for reconstruction, and 4.61 for expansion. A Gemini-based automatic evaluator assigns expansion mean scores of 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness. Expansion and reconstruction exhibit broadly similar interaction profiles; expansion's turn, overlap-event, backchannel, and interruption rates are 4.6%, 8.0%, 13.2%, and 16.0% lower, respectively. We evaluate data properties only; downstream gains in full-duplex model training remain for future work.