AnchorPrompt improves audio language model accuracy under noise and distortion
AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models
SoundMachine Learning
Summary
Audio-language AI systems can struggle to understand sound recordings that are noisy or distorted. The authors created AnchorPrompt, a method that teaches these AI models to stay consistent and avoid making things up when faced with unclear or corrupted audio. Instead of changing the whole model, they add a special prompt and train it to match the model’s answers on clean audio or to refuse answers when unsure. This helps the models handle new kinds of sound problems without extra preparation, improving reliability across tests.
What this means in practice
- •For voice assistant teams: Improve robustness of voice assistants to noisy or distorted records without retraining entire models.
- •For audio transcription services: Maintain accurate and reliable transcript outputs even under less-than-ideal audio conditions using frozen large models.
Authors
Pooneh Mousavi, Amir Ivry, Mirco Ravanelli, Cem Subakan
Abstract
Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.