Multi-speaker voice interaction benchmark reveals key challenges for ai agents
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Computation and Language
Summary
Voice assistants are becoming more common, but they often struggle when many people speak in the same setting, like meetings or homes. The authors created a new test called MSI-Bench to see how well AI understands and responds to multiple speakers talking at once. They found that existing systems have trouble figuring out who said what and when to respond, which makes voice interactions confusing. This work highlights specific problems to solve so future voice assistants can better handle group conversations.
What this means in practice
- •For meeting technology developers: Evaluate and improve AI systems' ability to understand and respond accurately to multiple speakers in real-time group meetings.
- •For smart home device engineers: Test voice assistants' capacity to correctly perceive and act on commands from multiple household members speaking together.
Authors
Chenxu Xiong, Dongming Shen, Yuzhi Tang, Wentao Ma, Mu Li, Alex Smola
Abstract
Voice provides a natural and immediate interface for AI agents. Many settings in which voice agents could be useful, including meetings, households, and collaborative work, are inherently multi-speaker. Supporting these settings introduces challenges that are largely absent from one-on-one interaction. We introduce the Multi-Speaker Interaction Benchmark (MSI-Bench) for evaluating multi-speaker voice interaction. Each test case is a short multi-party multi-turn audio scene with participant context, expected tool calls, and atomic rubrics. The benchmark targets three capability families: multi-speaker memory, multi-speaker instruction following, and multi-speaker reasoning. It comprises 1,152 test cases, evenly split between Mandarin Chinese and English (576 each). The strongest configuration on each split passes all rubrics on only 66.8% of English and 54.5% of Mandarin cases, and the strongest open-weight configuration on 34.0% and 19.3%. Failure analysis separates perception from reasoning: open-weight models are bottlenecked by the multi-speaker audio front-end, while frontier systems still fail speaker-scoped decision making on clean transcripts---and models across the board often respond when no one has addressed them. These results identify speaker-grounded perception, speaker-scoped decision making, and conversational restraint as concrete targets for future voice agents.