Full-duplex speech model handles long bilingual group conversations
MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation
Computation and LanguageArtificial IntelligenceSound
Summary
Understanding and responding to group conversations that last a long time and include multiple people speaking different languages is hard for current AI speech systems. The authors created a large synthetic dataset that mimics these complex conversations and a real-world benchmark to test models on long, multi-person, bilingual dialogues. They trained a new speech model that can follow and interact in these conversations more accurately than previous open-source systems. This work pushes AI closer to handling realistic meeting or social interaction scenarios involving English and Chinese speakers.
What this means in practice
- •For conference system developers: Build meeting transcription systems that maintain context and speaker roles in long bilingual group discussions.
- •For smart home assistant makers: Create voice assistants that understand and manage conversations among multiple household members speaking English and Chinese.$Commercial implications: Enables home assistants to handle complex, multi-user bilingual dialogues, improving interaction quality and usability.
Authors
Ke Wang, Houxing Ren, Zimu Lu, Yunqiao Yang, Zhuofan Zong, Mingjie Zhan, Hongsheng Li
Abstract
End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data ($\href{https://huggingface.co/datasets/MultiTalk/MultiTalkPT}{MultiTalkPT}$ and $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkFT}{MultiTalkFT}$) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce $\href{https://huggingface.co/datasets/MultiTalk/MultiTalkBench}{MultiTalkBench}$, built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.