Whisper encoder shrinks six layers with label-free recovery
Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Computation and LanguageMachine Learning
Summary
Large speech recognition models can be slow because they have many layers of processing. The authors found a way to remove six of the least important layers from the Whisper model's encoder without needing extra labeled data. They used speech data without transcripts to help the smaller model learn to perform nearly as well as before. This makes Whisper faster and simpler to run without changing how it is used.
What this means in practice
- •For speech recognition engineers: Deploy faster Whisper ASR models by removing less important encoder layers without custom inference code.
- •For mobile app developers: Integrate smaller Whisper models for quicker transcription on devices with limited computing power.
Authors
Rasmus Aagaard, Nicki Skafte Detlefsen
Abstract
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to $20.1\%$ after distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).