DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech
2026-08-17 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created DuplexGen, a new way to make synthetic conversations that sounds more natural. Instead of just writing dialogue and adding interruptions later, their system first writes the script, then two AI voices talk to each other in real-time, allowing natural timing and overlaps. Finally, a speech model reads the conversation without changing its timing. They tested this by making a patient-clinician conversation dataset and found their method better matches real dialogue timing than older methods.
dialogue synthesisfull-duplex communicationlarge language model (LLM)text-to-speech (TTS)conversational timingoverlapbackchannelsspeech corpusspeaker activityinteraction events
Authors
Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki
Abstract
Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dialogue synthesis pipelines typically generate dialogue content first and then insert interruptions, overlap, and backchannels using handcrafted markers or timing rules, making conversational timing prescribed rather than interaction-driven. We present DuplexGen, a dialogue synthesis framework that explicitly decouples content, timing, and acoustics. An LLM first generates the dialogue script, and then two full-duplex conversational models perform the script while listening to each other in real time. This allows conversational timing to emerge naturally while preserving the scripted content. Finally, a high-fidelity text-to-speech model re-renders the interaction without altering its timing. As a demonstration of the proposed framework, we construct a patient--clinician conversational speech corpus with construction-time annotations, including word timestamps, speaker activity, overlap regions, and interaction events. Experimental results show that the proposed framework produces conversational dynamics closer to real dialogue than conventional stitching-based synthesis.