Voice agents struggle to join group conversations naturally
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
Artificial IntelligenceComputation and Language
Summary
Voice agents today are good at talking with one person, but they find it very hard to join conversations with multiple people. The authors created a new test called MP-Bench to see how well these agents understand when to speak and how to respond correctly in group talks. They tested 12 popular voice agents and found that they perform poorly, almost guessing when to speak and understanding very little of the conversation in real time. This shows that more work is needed for voice assistants to work well in group settings.
What this means in practice
- •For voice assistant developers: Evaluate and improve voice agents’ ability to participate naturally in group dialogues using the MP-Bench benchmark.
- •For customer support teams: Design voice agents that better handle dynamic multi-party conversations in customer service environments.
Authors
Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang, Siddhant Arora, Shu-wen Yang, Abdelrahman Mohamed, Shinji Watanabe, Hung-yi Lee, David Harwath
Abstract
Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations. Evaluating agents in these settings is fundamentally more challenging than in dyadic interactions due to the exponentially greater conversational complexity. For voice agents to integrate seamlessly into human group dynamics, they must not only generate contextually appropriate responses but also demonstrate a nuanced understanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP-Bench), the first benchmark specifically designed to objectively evaluate conversational speech systems as active participants within multi-party contexts. MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness. Additionally, we incorporate comprehension-based question-answering tasks as a complementary evaluation. By benchmarking 12 voice agents, we find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking, exposing an open challenge for real-time voice agents under multiparty scenario.