Voice agents struggle to capture and fix spoken entity errors exactly
$τ$-Elicitation: Benchmarking multi-turn entity extraction in voice agents
Artificial Intelligence
Summary
Voice assistants often fail to capture exact details like names, dates, or addresses when you talk to them. The authors created a benchmark with 200 voice tasks to test how well these systems can understand and confirm such details. They found that while text-based systems get everything right, voice agents vary widely in success and often fail to fix mistakes, especially with hard or unfamiliar words. Adding steps like spelling and confirming details helps improve accuracy but makes calls longer. The main challenge is how the voice agents decide to verify and correct the information they get.
What this means in practice
- •For voice assistant developers: Improve voice assistants’ ability to confirm and correct exact details like names and addresses during multi-turn conversations.
- •For call center automation teams: Design spoken language systems that handle error recovery and verification for better customer data capture over voice calls.
Authors
Soham Ray, Victor Barres
Abstract
Voice agents often need to collect names, addresses, identifiers, dates, and times exactly, yet end-to-end benchmarks obscure where capture fails. We introduce $τ$-Elicitation, a 200-task voice benchmark spanning 10 entity types, controlled difficulty, caller realisms, and three environments. A matched text agent passes all tasks, but four voice configurations achieve robust exact success from 0.14 to 0.41. Agents increase verification for hard and unfamiliar entities and sometimes for incorrect captures, but not for their weakest caller voice; only 24 to 37 percent of verified errors are repaired. A scaffold that prescribes spelling, read-back, correction, and confirmation raises robust Pass$^3$ by 14 to 31 points, at a cost of 21 to 28 seconds per call. Realisms such as spelling variations and restarts do not detectably affect exact success; mispronunciation increases repair effort. These results identify strategy selection and successful recovery as the central bottlenecks in exact spoken entity collection.