Audio models cut tokens to speed speech tasks with little accuracy loss
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
SoundArtificial Intelligence
Summary
Audio language models turn sounds into many pieces called tokens, which makes understanding speech slow because the processes consider every token with every other. Current methods to reduce tokens miss important speech changes, making it harder to recognize words accurately. The authors propose AudioTETRIS, which selects the important sound pieces based on changes in the audio and keeps those through the decoding process without needing retraining. This approach keeps nearly the same accuracy while reducing the number of tokens to a quarter, speeding up speech-related tasks.
What this means in practice
- •For speech technology developers: Reduce computation for speech recognition systems by compressing audio tokens without losing accuracy, enabling faster real-time processing.
- •For virtual assistant engineers: Improve response times of voice-enabled assistants by using token elimination methods to handle audio input more efficiently.
Authors
Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren
Abstract
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.