Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluation
SoundComputation and Language
Summary
The authors studied how well speech and audio language models work by comparing using raw audio (waveform) versus text transcripts. They set up a method to decide when to use transcripts, audio features, or advanced generative models to label sounds. They found that while raw audio helps, strong non-generative audio features alone perform nearly as well as using generative models. Adding generative calls gave only a tiny improvement, suggesting that for known tasks, it’s important to measure the extra benefit of generative audio models beyond transcripts and audio features. Their work helps clarify how to fairly evaluate these systems.
Authors
Mengzhe Geng
Abstract
Speech and audio LLMs are often evaluated by asking whether a waveform prompt beats an automatic speech recognition (ASR) transcript. For known closed-set tasks, that comparison conflates two factors: access to acoustic evidence and the need to call a generative audio model. We evaluate this distinction as a controlled call-decision problem. For each example, a policy chooses among keeping a transcript label, using encoder evidence from Contrastive Language-Audio Pretraining (CLAP), Audio Spectrogram Transformer (AST), or WavLM, and calling Qwen2-Audio, Qwen2.5-Omni, or MOSS-Audio; the decisive ablation removes all generative actions while keeping the selector and development protocol fixed. On VocalSound, transcripts reach 0.296 accuracy, so waveform information is needed. Yet supervised CLAP and WavLM controls reach 0.850 and 0.854 with no generative audio calls. A selector with generative actions reaches 0.925 accuracy using 12.5% calls, compared with 0.921 for the matched no-call selector (paired difference 0.004; 95% CI [-0.025,0.033]). Agreement and stacking features improve weaker selectors but do not beat the strongest no-call control. For known-task endpoint claims, the relevant quantity is the marginal value of the generative call after transcript and encoder evidence have already been used.