Streaming speech recognition system reduces delay while improving accuracy
X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR
SoundArtificial Intelligence
Summary
Real-time speech recognition systems need to quickly and accurately convert spoken words into text without long pauses. The authors propose a new approach called X2Streaming-ASR that decides when to finalize parts of the transcript based on how confident it is, rather than using fixed time chunks. They train the system in stages to balance speed and accuracy better than previous methods. Their approach significantly cuts the delay between hearing speech and displaying the text while keeping errors low across multiple standard datasets.
automatic speech recognitionstreaming ASRlatencycommit policyreal-time processingcharacter error rateforced alignmentacoustic boundariestraining procedure
Authors
Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
Abstract
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across AISHELL-1/2/3 and WenetSpeech, X2Streaming-ASR achieves a mean character-level commit latency of 27-84 ms relative to forced-aligned character endpoints, compared with 409-585 ms for the evaluated streaming baselines. It achieves the best streaming CER among the evaluated systems on AISHELL-1 and AISHELL-3 with substantially lower latency.