Audio scorer improves large audio models by picking key sounds efficiently
AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models
SoundComputation and Language
Summary
Listening to long audio clips like speeches or music requires a lot of computing power because the computer has to process many sound segments. The authors created AudioScorer, a small tool that picks out the most important parts of the audio based on the question someone asks. It learns which sound bits matter most by watching how a big audio model focuses on answers. This makes answering questions from long audio faster and still accurate for tasks like recognizing speech or emotions.
What this means in practice
- •For voice assistant developers: Increase speed and reduce costs of processing long audio queries by selecting only relevant sound tokens per user question.
- •For audio analytics teams: Improve accuracy and efficiency in tasks like emotion and scene recognition by focusing on question-relevant audio snippets.
Authors
Marwan Farag, Bin Yang
Abstract
Large Audio Language Models (LALMs) reason over speech, environmental sounds, music, and other acoustic information. However, their audio encoders produce dense token sequences that make long-form audio costly to process. Existing audio pruning methods reduce this cost using heuristics such as attention, representation similarity, or temporal sampling, but do not explicitly learn which tokens are most useful for a given question. We propose AudioScorer, a lightweight model that scores audio tokens according to the user question. It is trained by distilling the ranking of audio tokens under the frozen LALM's answer-gradient saliency, together with downstream answer supervision. Compared with other pruning methods, AudioScorer achieves the best or tied-best result at 10% token retention on 7 of 10 semantic and acoustic tasks, including speech recognition, emotion recognition, and audio-scene classification.