Audio language models struggle to ignore irrelevant speakers near voice assistants
Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents
SoundArtificial IntelligenceComputation and LanguageMultimedia
Summary
Voice assistants need to decide when to respond to spoken commands, especially when many people talk nearby. The authors created a test called VGBench to check if audio language models can tell when to act or stay silent based on who is speaking and the situation around them. They found that current models often respond even when they shouldn't, like reacting to a bystander instead of the user. The authors improved this by training a system called VoxGate, which helps the assistant correctly ignore commands from others while still responding to the user.
What this means in practice
- •For voice assistant developers: Use VGBench and VoxGate to improve voice assistants so they respond only to the intended user and ignore background speakers.
- •For smart home device makers: Enhance devices to better detect when commands come from the primary user versus others nearby, reducing accidental activations.
Authors
Yanjie Zhang, Nanchen Hu, Yushi Sun
Abstract
Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.