Audio transformers run faster with smarter token reduction methods
Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers
SoundMachine Learning
Summary
Processing audio with transformer models can be slow and costly because these models look at many small parts of sound called tokens. The authors studied ways to cut down the number of tokens the model processes without losing accuracy, especially by using information about the sound’s frequency. They found that methods that consider the audio’s frequency patterns work better than those using only the model’s attention signals. This helps make audio models faster and more efficient, particularly when handling long sounds.
What this means in practice
- •For audio software developers: Build more efficient audio recognition tools that use fewer sound tokens while keeping accuracy high by using frequency-aware token reduction.
- •For speech recognition engineers: Improve the speed of large-scale speech transcription systems by integrating token sparsification based on audio frequency features.
Authors
Shreya Sajid, Fabrizio Julian Piva, Hiram Rayo Torres Rodriguez, Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin Vanschoren
Abstract
Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.