X2Streaming-ASR cut delay drastically in live speech recognition
X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR
SoundArtificial Intelligence
Summary
Real-time speech recognition systems often struggle to balance quick response times with accurate partial transcripts. The authors propose a new method called X2Streaming-ASR that decides when to commit to recognizing spoken words and what to commit in a more flexible way. Their approach reduces the delay between hearing and transcribing speech from hundreds of milliseconds to just tens, while improving accuracy on challenging datasets. This could help voice assistants and dialogue systems respond faster and more accurately.
What this means in practice
- •For voice assistant developers: Implement faster, more accurate speech recognition for live voice agents with lower transcription delays.
- •For real-time dialogue system builders: Improve full-duplex chatbot responsiveness by committing transcript outputs only when confident, enhancing user experience.
Authors
Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
Abstract
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across AISHELL-1/2/3 and WenetSpeech, X2Streaming-ASR achieves a mean character-level commit latency of 27-84 ms relative to forced-aligned character endpoints, compared with 409-585 ms for the evaluated streaming baselines. It achieves the best streaming CER among the evaluated systems on AISHELL-1 and AISHELL-3 with substantially lower latency.