Multi-party speech systems struggle with interruption detection
MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue
SoundArtificial IntelligenceComputation and Language
Summary
Deciding who should talk and when in group conversations is hard for speech systems. The authors created MultiPartyTurnBench, a test set to measure how well computers detect when someone finishes speaking or interrupts in a multi-person setting. They found these systems are good at spotting when a speaker’s turn ends but poor at recognizing interruptions or deciding who should yield the floor. This reveals that smart turn-taking in groups remains a difficult problem for voice-based technology.
What this means in practice
- •For voice assistant developers: Improve multi-party voice assistants by refining their ability to detect when one speaker finishes and another interrupts, enhancing natural interactions.
- •For call center technology teams: Develop better tools for managing multi-speaker calls by recognizing true interruptions and floor control, reducing confusion in automated systems.
Authors
Puneet Mathur, Dinesh Manocha
Abstract
Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.