Audio token attention can be predicted before language model runs
Audio Token Attention Is Predictable Before the Language Model Runs
SoundArtificial IntelligenceComputation and Language
Summary
Understanding which parts of audio are important can help process speech faster and fit more audio into a model's limited memory. The authors found that how much attention audio pieces get inside a large language model can be predicted early on, even before the model fully processes the audio. This lets them cut down on less important audio parts early, speeding up transcription without losing much accuracy. Their method called Triage compresses audio input efficiently and works better than other known methods.
What this means in practice
- •For speech recognition engineers: Speed up real-time speech transcription systems by reducing audio input size without losing accuracy using attention prediction before model execution.
- •For cloud service operators: Increase the number of simultaneous speech transcription streams a single GPU can handle by applying early audio token pruning.
Authors
Kyoungjun Park, Yunzhe Li, Lili Qiu
Abstract
A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at $ρ\geq .69$ on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io