Controlled synthesis improves full-duplex conversation turn taking
A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations
Artificial Intelligence
Summary
Full-duplex dialogue systems can listen and speak at the same time but struggle to tell when someone has finished speaking, paused, or wants to interrupt. The authors developed a method to create realistic conversation audio with clear labels for these different speaking events, using a large language model to generate dialogue turns and intents without fixed timing. Their system produces speech that covers many speaking behaviors and languages, which helps train full-duplex models to manage turn-taking more accurately. After training on this synthetic data, a conversation model improved its ability to know when to start and stop speaking.
What this means in practice
- •For voice assistant developers: Train dialogue systems that better handle interruptions and pauses using labeled, synthetic conversational speech data.
- •For telecommunications engineers: Improve real-time conversational floor control in communication devices with more accurate turn-taking models trained on diverse synthetic dialogues.
Authors
Matthew Sun, Vinay Kothapally, Meng Yu, Chao Huang, Hao Zhang, Yixuan Zhang, Steve Yves
Abstract
Full-duplex dialogue systems, which listen while speaking, must distinguish a completed turn from a pause within a turn and an interruption that requests a turn from a brief acknowledgment or speech addressed to a third party. Yet existing conversational corpora provide limited control over these events and limited labels for their intent. We present a pipeline for synthesizing intent-labeled, two-channel conversational speech from relational event lists. An LLM authors each event's speaker, text, conversational act, and attachment to an earlier event without predicting absolute timestamps. Events are synthesized independently, aligned with their source text, and placed on a shared clock, so turn-taking landmarks are measured from the rendered signal while silence durations are specified or sampled from turn-taking distributions. The pipeline covers 42 phenomena across eight families in English and Mandarin, derives frame-level system actions from authored intent, and promotes diversity using small, diverse sets of prior examples and batch prompts that request alternatives with self-reported probabilities. Ablations show gains in each targeted diversity dimension. On a four-action label space for taking, holding, releasing, and not holding the conversational floor, a semantic voice-activity detector using only current and past audio reaches start-speaking and start-listening F1 scores of 0.819 and 0.802. When generating its own responses, the full-duplex speech model Moshi takes 0.85 of the reference turns after fine-tuning on the generated corpus, compared with 0.44 before fine-tuning. Its frame-level precision for predicting system-floor occupancy rises from 0.46 to 0.88. With reference context at each step, its frame-level floor F1 rises from 0.893 to 0.962. These results show that controlled synthesis can provide learnable and transferable supervision for full-duplex turn management.