Lossless method speeds up speech recognition by nearly two times
Lossless Inference Acceleration for ASR via EAGLE-Style Speculative Decoding
Sound
Summary
Automatic speech recognition models are very good but slow because they generate words one at a time. The authors introduce a new lightweight draft system called Speech-Draft that guesses multiple words ahead, helping the main model work faster without making mistakes. This new method, called EAGLE-ASR, speeds up recognizing speech by up to 2.4 times while keeping perfect accuracy. It works well on different popular speech models and even supports multiple languages and speech translation.
What this means in practice
- •For speech technology engineers: Implement faster speech-to-text systems that maintain perfect accuracy by integrating EAGLE-ASR speculative decoding.
- •For language service providers: Deploy multilingual and speech translation services with reduced latency without losing transcription quality.
Authors
Sen Zhang, Xianghu Yue, Yiming Chen, Wenhuan Lu, Jianguo Wei, Haizhou Li
Abstract
Autoregressive Transformer and Conformer models have achieved state-of-the-art performance in automatic speech recognition (ASR). However, their sequential token generation mechanism imposes a significant bottleneck during decoding, resulting in substantial inference latency. While speculative decoding has emerged as a promising lossless acceleration technique, existing ASR-oriented approaches either suffer from low draft acceptance rates due to suboptimal draft quality or rely on specially designed external draft models, which limit their generality and practicality. To overcome these limitations, we propose EAGLE-ASR, a novel framework that accelerates ASR inference via EAGLE-style speculative decoding while strictly preserving lossless output quality. Specifically, we introduce Speech-Draft, a lightweight autoregressive module that serves as a universal draft model by jointly leveraging encoder acoustic features and hierarchical decoder representations from the target ASR model. Furthermore, we adopt a multi-step generation strategy with self-distilled data to train the Speech-Draft, enabling it to better approximate the target model's output distribution and maintain causal consistency during multi-token generation. Extensive experiments on representative Whisper and Conformer AED models demonstrate that EAGLE-ASR achieves significant lossless speedups of up to $1.9\times$ and $2.4\times$, respectively, consistently outperforming existing speculative decoding baselines for ASR. Our proposed framework further generalizes effectively to multilingual ASR and speech translation tasks, establishing EAGLE-ASR as a practical and universally applicable solution for accelerating modern ASR systems without compromising recognition accuracy.