Full-duplex speech models add live user transcription with low delay
Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models
Computation and Language
Summary
Full-duplex speech-to-speech models can listen and speak at the same time, making conversations with AI more natural. But they usually don’t have a built-in way to show what the user is saying word-for-word. The authors found a way to add live transcription of the user's speech to these models by attaching a small, efficient component. This lets the system write down what users say as they talk without slowing things down or changing how the conversation flows. Their method performs well compared to existing transcription models and keeps the conversation feeling natural.
What this means in practice
- •For conversational ai developers: Add live user speech transcription to full-duplex speech systems without redesigning core models, enabling real-time conversation logging and monitoring.
- •For accessibility software teams: Provide instant captions for user speech in interactive AI systems to support users with hearing impairments during natural conversations.
Authors
Ke Hu, Nourchene Ferchichi, Edresson Casanova, Ankita Pasad, Elena Rastorgueva, Chen Chen, Nithin Rao Koluguri, Piotr Zelasko, Yifan Peng, Hainan Xu, Zhehuai Chen, Boris Ginsburg
Abstract
Full-duplex speech-to-speech (S2S) models enable natural conversational AI by allowing simultaneous listening and speaking. However, these models typically lack inherent user speech transcription, which is essential for applications such as conversation logging, accessibility features, and quality monitoring. In this work, we propose an efficient method to add streaming ASR capabilities to an existing duplex S2S model by introducing a lightweight ASR head in parallel to the agent text head. Our approach requires minimal additional parameters and no significant architectural changes to the base S2S model, enabling real-time user transcription while preserving full-duplex conversational capabilities including turn-taking and barge-in handling. Experimental results demonstrate that our method achieves streaming average WER of 10.21% on the HuggingFace Open ASR Leaderboard within the duplex S2S framework. Additionally, we show that the same architecture trained as a standalone streaming ASR model achieves competitive results (7.73% WER) compared to current SOTA models. We will open-source our training and inference code to facilitate further research in joint streaming ASR and S2S modeling.