Efficient streaming text formatting from speech using pretrained models
Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy
Computation and Language
Summary
When computers listen to speech and turn it into text, they often write it exactly as spoken, which can be hard to read. The authors developed a new method to quickly and accurately change the spoken-style text back into normal written style while the speech is still coming in. They used a powerful existing language model and changed it so it can handle input as it arrives without waiting for the entire sentence. Their system works well in Vietnamese, matching slower methods but acting fast enough for real-time use.
What this means in practice
- •For speech recognition engineers: Convert live speech-to-text outputs into readable written format in real time for improved user interfaces.
- •For natural language processing teams: Deploy streaming-friendly text normalization for various languages without needing handcrafted rules, enabling easier scaling.
Authors
Kiet Anh Hoang, Khanh Le, Bao Nguyen, Linh Pham, Dung Vo, Thai Tran, Mai Nguyen, Tri Nguyen, Vu Le
Abstract
Streaming Inverse Text Normalization (ITN) is vital for converting spoken-form outputs from streaming Automatic Speech Recognition into formatted written text. Existing streaming ITN methods rely on hybrid systems combining neural tagging with expert-crafted finite-state transducer rules, limiting scalability across domains and languages. While end-to-end models offer superior scalability by learning directly from data, standard encoder-decoder architectures are inherently non-streaming due to global attention. In this paper, we propose an efficient streaming end-to-end ITN system, adapting a pretrained text-to-text model to leverage its robust linguistic knowledge. To enable streaming, we introduce architectural adaptations, a specialized training strategy, a Read-Tag-Write decoding policy, and inference optimizations. Experiments on Vietnamese datasets show accuracy comparable to non-streaming baselines, outperforming hybrid methods while satisfying real-time latency requirements.