Streaming speech recognition balances accuracy speed and memory for multiple speakers
Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs
Sound
Summary
Recognizing multiple people talking at the same time in real time is hard because the system has to be fast, accurate, and remember long conversations. The authors studied four different ways to combine identifying who is speaking with transcribing what is said, using the same basic speech and speaker models. They tested how well each way works in accuracy, memory use, and training difficulty. Their study helps clarify which approach is best depending on the situation when building systems that listen to many speakers live.
What this means in practice
- •For call center technology teams: Build real-time tools that transcribe and separate voices in customer service calls to improve understanding and response accuracy.
- •For conference system engineers: Deploy streaming ASR systems that handle multiple overlapping speakers with controlled memory use for live meeting transcription.
Authors
Taejin Park, Ivan Medennikov, Kunal Dhawan, Weiqing Wang, Jagadeesh Balam, Boris Ginsburg
Abstract
Streaming multi-speaker ASR is a challenging task that must balance accuracy, latency, and efficiency while handling overlapping speech and maintaining coherent long-context modeling over extended conversations in an online fashion. We present a unified framework that categorizes streaming multi-speaker ASR into four architectural strategies based on how diarization and ASR are integrated. Using a shared pair of open-source streaming ASR and diarization models as a common foundation, we derive four multi-speaker ASR systems that differ in whether they employ multiple model instances, fine-tuning, or both. We evaluate these systems across multi-speaker accuracy, single-speaker accuracy degradation, memory footprint, and training complexity. Through this systematic architectural analysis, we clarify the design space for streaming multi-speaker ASR and provide practical guidance for selecting the most suitable approach under diverse deployment constraints.