Uni-Stream unifies real-time speech recognition and synthesis in one model
Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition
Artificial Intelligence
Summary
Converting speech to text and text to speech usually requires separate systems, which can slow down real-time conversations with computers. The authors present Uni-Stream, a single model that can handle both tasks at the same time without losing quality. It processes speech and text together as they come in and uses a special two-step training method to get good performance. They also created a method to label data with word-by-word timings and will share these annotations publicly. Their experiments show that even though the model uses one architecture, it keeps speech and text information separate internally while adapting to each task.
What this means in practice
- •For voice assistant developers: Build voice assistants that can simultaneously understand spoken commands and respond with natural-sounding speech using a single unified model.$Commercial implications: Enables development of commercial voice assistants with integrated recognition and synthesis, reducing latency and complexity.
- •For call center technology teams: Create real-time transcription and response systems using one model that handles both speech input and output more efficiently.
Authors
Xuan Shi, Chang Zeng, Tiantian Feng, Ye Zhang, Jun Liu, Shih-Heng Wang, Lie Lu, Shrikanth Narayanan
Abstract
Unified streaming speech language models aim to support speech recognition and synthesis within a single model, enabling real-time human-computer interaction. However, jointly supporting these two directions of speech-text conversion while preserving task-specific performance remains challenging. We propose Uni-Stream, a unified streaming speech language model built on compact pretrained speech models. Uni-Stream processes interleaved text and speech in a streaming format and achieves performance comparable to task-specific models through two-stage training. We also develop a data annotation pipeline for streaming training and will publicly release the generated transcriptions and word-level timestamps. We further examine Uni-Stream's representations across tasks and modalities and find that speech and text residence in distinct latent feature space even when trained in a unified model architecture, while representations within each modality become more task-dependent in later layers.