DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios
2026-08-24 • Sound
Sound
AI summaryⓘ
The authors developed a new model called DiaScriber to recognize what multiple people say, who said it, and when they said it in conversations with overlapping or fast speaker changes. They created diverse training data to represent many real-world speaking situations and trained their model in three stages using a large language model as a base. Their experiments showed that DiaScriber works better than existing methods and can handle new, unseen multi-speaker conversations well. This approach helps with automatic speech recognition involving multiple speakers in complex scenarios.
multi-speaker automatic speech recognitionspeaker diarizationoverlapping speechturn transitionspeech large language modelpretrained modelfine-tuningreinforcement learningdata simulationmulti-speaker scenarios
Authors
Bingshen Mu, Xian Shi, Xiong Wang, Zhifang Guo, Ting He, Xize Cheng, Yu Xi, Jin Xu, Lei Xie
Abstract
Multi-speaker automatic speech recognition (MSASR) aims to jointly predict content transcriptions, speaker identities, and timestamps, thereby addressing the key question of "who spoke what and when" and holds substantial practical value in real-world multi-speaker scenarios. However, MSASR still encounters considerable challenges in the presence of fast turn transitions, overlapping speech, and complex, diverse multi-speaker scenarios. In this work, we propose DiaScriber, an end-to-end multi-speaker diarization and transcription model built on a speech large language model. We first construct diverse data pipelines to cover a wide variety of multi-speaker scenarios and their complexities, including validation and refinement, turn-transition and overlapping-speech simulation, and multimodal annotation. Furthermore, DiaScriber is developed based on the pretrained version of Qwen3.5-Omni through a three-stage training strategy involving continual pretraining, supervised fine-tuning, and reinforcement learning. Experiments show that DiaScriber achieves superior performance over comparison methods across extensive multi-speaker scenario test sets and demonstrates outstanding generalization ability in unseen multi-speaker scenarios.