Papers for

utility service providers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Voice agent benchmark reveals challenges handling calls with background speech

VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation

Abstract: Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark that combines these challenges in 100 calls about utility billing and payment assistance. Each caller makes two to four requests. The agent has sixteen tools backed by a stateful Stripe billing twin and the Apache Fineract loan engine, with account access blocked until caller verification succeeds. The tasks use public household electricity data and a policy based on Pennsylvania's residential billing rules. An LLM-as-a-verifier checks the agent's actions and spoken figures against explicit requirements. On a calibration run, it agrees with a code verifier on 99.1% of checks. Across fourteen voice stacks and three repeats per task, completion ranges from 17.3% to 44.7%. Grok Voice leads, and Gemini 3.8 Live and GPT-Live follow at about the same cost per call. Background television reduces pooled completion from 38.7% to 8.6%. The simulated caller often accepts an incorrect result because it hears the agent's words but cannot inspect its actions. These findings show why voice agents need evaluation across the whole call, including what they say, what they change, and how they handle competing speech.

Tue 29 SeptArtificial Intelligence
The gist
Handling phone calls with voice agents is tricky when there are multiple requests, background noise, and impatient callers. The authors created VAmoS Energy, a test system with 100 simulated calls about utility bills that mimic these real-world challenges. They found current voice agents often fail to complete tasks, especially when there is background TV noise, because callers can't check the agent’s actions—only hear its words. This work shows voice agents need thorough testing of both their spoken responses and their behind-the-scenes actions.
Open → 2609.38512v1