Papers for

accessibility software teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Full-duplex speech models add live user transcription with low delay

Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models

Abstract: Full-duplex speech-to-speech (S2S) models enable natural conversational AI by allowing simultaneous listening and speaking. However, these models typically lack inherent user speech transcription, which is essential for applications such as conversation logging, accessibility features, and quality monitoring. In this work, we propose an efficient method to add streaming ASR capabilities to an existing duplex S2S model by introducing a lightweight ASR head in parallel to the agent text head. Our approach requires minimal additional parameters and no significant architectural changes to the base S2S model, enabling real-time user transcription while preserving full-duplex conversational capabilities including turn-taking and barge-in handling. Experimental results demonstrate that our method achieves streaming average WER of 10.21% on the HuggingFace Open ASR Leaderboard within the duplex S2S framework. Additionally, we show that the same architecture trained as a standalone streaming ASR model achieves competitive results (7.73% WER) compared to current SOTA models. We will open-source our training and inference code to facilitate further research in joint streaming ASR and S2S modeling.

Mon 14 SeptComputation and Language
The gist
Full-duplex speech-to-speech models can listen and speak at the same time, making conversations with AI more natural. But they usually don’t have a built-in way to show what the user is saying word-for-word. The authors found a way to add live transcription of the user's speech to these models by attaching a small, efficient component. This lets the system write down what users say as they talk without slowing things down or changing how the conversation flows. Their method performs well compared to existing transcription models and keeps the conversation feeling natural.
Open 2609.15759v1