Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
2026-08-31 • Sound
SoundArtificial Intelligence
AI summaryⓘ
The authors study how Whisper, a speech processing model, uses a long sequence of tokens to understand audio, which may include unnecessary information. They introduce a method called stride-k subsampling that keeps only every k-th token to reduce this redundancy without much loss in accuracy. Testing this on different sizes of Whisper showed that cutting tokens by half or more significantly lowers computing costs and speeds up processing with only small drops in recognition performance. This approach works well in both speech-to-text tasks and speech language models, requiring no extra training. Overall, the authors show that Whisper’s token representation can be simplified to make processing more efficient.
Whisperautomatic speech recognition (ASR)encoder tokensstride-k subsamplingword error rate (WER)convolutional neural networktransformercanonical correlation analysis (CKA)GFLOPsspeech language model (SpeechLM)
Authors
Chanhee Cho, Junhyuk Choi, Bugeun Kim
Abstract
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.