Voice agent benchmark reveals challenges handling calls with background speech
VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation
Artificial Intelligence
Summary
Handling phone calls with voice agents is tricky when there are multiple requests, background noise, and impatient callers. The authors created VAmoS Energy, a test system with 100 simulated calls about utility bills that mimic these real-world challenges. They found current voice agents often fail to complete tasks, especially when there is background TV noise, because callers can't check the agent’s actions—only hear its words. This work shows voice agents need thorough testing of both their spoken responses and their behind-the-scenes actions.
What this means in practice
- •For voice assistant developers: Test and improve voice agents by simulating realistic multi-request calls with background noise and verifying responses against complex billing tasks.
- •For call center technology teams: Use VAmoS Energy to benchmark and compare voice system performance under noisy conditions that reflect customer impatience and account verification challenges.
- •For utility service providers: Evaluate voice-enabled billing systems using simulations that incorporate real regional billing policies and stateful account operations for realistic training.$Commercial implications: Enables sale of specialized voice assistant testing platforms tailored to utility companies needing compliance and reliability assurance.
Authors
Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi
Abstract
Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark that combines these challenges in 100 calls about utility billing and payment assistance. Each caller makes two to four requests. The agent has sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with account access blocked until caller verification succeeds. The tasks use public household electricity data and a policy based on Pennsylvania's residential billing rules. An LLM-as-a-verifier checks the agent's actions and spoken figures against explicit requirements. On a calibration run, it agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats per task, completion ranges from 17.3% to 44.7%. Grok Voice leads, and Gemini 3.8 Live and GPT-Live follow at about the same cost per call. Background television reduces pooled completion from 38.7% to 8.6%. The simulated caller often accepts an incorrect result because it hears the agent's words but cannot inspect its actions. These findings show why voice agents need evaluation across the whole call, including what they say, what they change, and how they handle competing speech.