Voice agent reliability measured by inquesto score protocol
Inquesto Score: A reliability Protocol For Voice Agents
Computation and LanguageArtificial Intelligence
Summary
Voice agents like digital assistants sometimes fail to finish tasks, which can cause problems during important interactions. The authors created the Inquesto Score to measure how reliable these agents are by checking if calls reach the user's goal without errors or delays. Their method looks directly at audio issues and checks the conversation's meaning and state to find mistakes. They tested this score on various voice agent setups and show it needs more than just transcripts to work well. They provide all their tools and data so others can use and improve this scoring method.
What this means in practice
- •For voice assistant developers: Assess voice assistant reliability in diverse acoustic and user scenarios to improve deployment safety and user trust.
- •For customer experience teams: Identify and diagnose specific failure types in voice agents to enhance call quality and reduce user frustration.
Authors
Massa Baali, Bhiksha Raj
Abstract
Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creating a need for reproducible and interpretable evaluation. We introduce Inquesto Score (IS), a protocol for measuring voice-agent reliability as the percentage of calls in a fixed, versioned evaluation population that achieve the caller's goal without a functional failure or worse. Rather than combining heterogeneous metrics, IS defines explicit failure events and severity levels and evaluates the deployed voice pipeline. Timing failures, including talk-over and delayed responses, are measured directly from audio, while semantic and state-dependent failures are evaluated using scenario predicates, tool traces, and a pinned open-model judge. Diagnostic views of behavior, acoustic robustness, identity handling, and speaker groups accompany the score without being combined into it. Inquesto Score v0.1 evaluates 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations of a reference voice-agent system. Our evaluation shows that reliable measurement requires evidence beyond transcripts, explicit treatment of deployment conditions, and validation of the evaluators used to determine outcomes. We release the protocol, reference implementation, and evaluation records.