Selective control reduces audio interference in large multimodal models
Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models
SoundComputation and LanguageMachine Learning
Summary
Large audio-language models listen to sound and text together, but sometimes unrelated sounds confuse their answers. The authors found that controlling how much the model listens to audio can improve its reasoning without hurting speech recognition. They designed a method called ICAP-Gate that selectively adjusts audio influence based on the task. This method works better than previous fixes and keeps the model fast and accurate when handling mixed sounds.
What this means in practice
- •For multimodal ai developers: Build AI systems that adjust audio input influence for more reliable reasoning under noisy conditions.
- •For voice assistant engineers: Improve voice assistants by preserving speech recognition accuracy while reducing errors from irrelevant background sounds.
Authors
Yulin Sun, Kele Xu, Yong Dou
Abstract
Large audio-language models (LALMs) exploit multimodal evidence, yet task-irrelevant audio can alter text-reasoning decisions when listening is unnecessary. Aggregate Accuracy can hide this paired drift because audio-induced repairs and damages may cancel. Paired drift analysis and targeted interventions identify architecture-specific, intervention-sensitive late audio pathways as actionable control points. We introduce ICAP-Gate, which applies mechanism-guided, task-conditioned control to each model's pathway. Across four LALMs, two reasoning benchmarks, and environmental-sound and natural-speech interference, ICAP-Gate has lower point estimates for Influence Rate and Answer Flip than ungated inference in all 16 full-split model--condition evaluations. Fixed suppression degrades automatic speech recognition (ASR) across all four models, whereas ICAP-Gate matches ungated ASR performance by preserving the pathway for explicit audio-demand instructions. ICAP-Gate has lower paired-drift point estimates than mitigation prompting in all four evaluated settings and provides competitive stabilization relative to eight-sample Self-Consistency while using one generation per query; in controlled ARC measurements, Self-Consistency incurs $7.0$--$9.2\times$ ungated latency. These results establish selective modality influence control as a design principle for robust multimodal reasoning.