Streaming speech recognition improves speed and accuracy with new training method
Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR
SoundArtificial Intelligence
Summary
Transcribing speech as it is spoken needs to balance accuracy with how quickly words appear. The authors found that existing methods use a timing measure that doesn’t match what users actually notice and limits how early the system can guess words. They introduced a new way to measure delays and trained the system to optimize both accuracy and delay at the same time. This approach made the speech recognizer both faster and more accurate without changing its basic design.
What this means in practice
- •For speech recognition developers: Improve real-time speech transcription by jointly optimizing accuracy and word emission speed without changing model architecture.
- •For live captioning providers: Reduce caption delays while increasing transcription accuracy for streaming audio using new latency-aware training techniques.
Authors
Xiuwen Zheng
Abstract
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $τ$ that bounds the decoder's lookahead. We show that $τ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $τ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($τ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.