Factorized model improves streaming speech recognition with less memory
Factorized Delayed Streams Modeling for LLM-based Streaming ASR
Computation and LanguageSound
Summary
Automatic speech recognition (ASR) that understands spoken words as they happen can be tricky because the computer needs to line up sounds with text quickly. The authors looked at a way to improve such systems by changing how they handle extra tokens that mark waiting times during transcription. They showed that removing one of these tokens doesn't hurt performance and then proposed a new method that separates the waiting signal from actual word predictions. This new approach uses less memory, runs faster, and improves accuracy compared to the older method.
What this means in practice
- •For speech recognition engineers: Reduce memory use and speed up streaming ASR systems while improving recognition accuracy in live transcription scenarios.
- •For mobile app developers: Build voice-controlled live apps that consume less device memory and provide faster and more accurate speech-to-text responses.
Authors
Tatsunari Takagi, Kai Washizaki, Atsushi Kojima, Lianbo Liu, Koki Nikaido, Yui Sudo
Abstract
Delayed Streams Modeling (DSM) enables LLM-based streaming automatic speech recognition (ASR) by aligning acoustic and text streams on a common timeline. DSM adds the padding token <p> and the word-start token <w> to the LLM vocabulary and predicts them together with normal text tokens using the same softmax. We first show that <w> can be removed while maintaining competitive recognition performance. Based on this result, we propose Factorized DSM (F-DSM), which separates the waiting probability for <p> from the distribution over the original LLM vocabulary. This factorization removes ASR-specific tokens from the text prediction space and allows the large-vocabulary softmax to be skipped on waiting steps. Experiments on the Corpus of Spontaneous Japanese and LibriSpeech show that F-DSM achieves better recognition performance than DSM. It also greatly reduces GPU memory use while maintaining similar training throughput, provides a small inference speed improvement through softmax skipping, and reduces the degradation in text-only perplexity observed with DSM.