Voice interfaces tested for reliability with computer-use agents
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Artificial Intelligence
Summary
Talking to computer programs using your voice is becoming more common, but speech recognition can mess up important details when converting spoken words into text. The authors created Talk2Agent, a way to check how well voice systems pass on spoken instructions to computer programs that use large language models. Instead of always running tasks to see if they work, their method evaluates whether the key information from speech was kept correctly, saving time and effort. Their approach better predicts how well the tasks will be done compared to traditional speech error metrics.
What this means in practice
- •For voice interface developers: Evaluate and improve how speech input retains task-critical details for AI assistants without costly task executions.
- •For software product managers: Measure voice system quality in real-world conditions to select better speech technology for applications using language model-based agents.
Authors
Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang
Abstract
Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.