HINTT compares cascaded and unified speech models for multilingual diarization and transcription
HINTT Submission to the 2nd MLC-SLM Challenge: Comparing Cascaded and Unified Approaches to Diarization and ASR
Computation and Language
Summary
Understanding who said what and when in a conversation across multiple languages is a tricky problem. The paper describes two ways to solve this: one that breaks the task into steps (finding speakers first, then transcribing), and another that tries to do everything at once with a single model. The authors find that, for this challenge, their step-by-step method works more reliably, but the all-in-one approach shows promise for the future.
What this means in practice
- •For call center operators: Use the cascaded speech diarization and ASR system to reliably attribute speech and transcriptions to multiple speakers in multilingual customer calls.
- •For conference transcription teams: Deploy unified speech language models to generate speaker labels, timing, and transcripts in multilingual meetings, simplifying pipeline complexity as models improve.
- •For multilingual voice assistant developers: Incorporate the stepwise diarization plus ASR approach enhanced by language model error correction to improve speaker-aware transcription accuracy in mixed-language environments.$Commercial implications: Enables advanced multilingual voice assistants that correctly identify speakers and transcribe speech, a key differentiator for commercial products.
Authors
Takanori Ashihara, Kohei Matsuura, Masato Mimura
Abstract
This paper presents the HINTT system submitted to the 2nd Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM). We address multilingual speaker-attributed ASR, where systems must determine who spoke when and what was spoken. We investigate two modeling strategies for this problem: a cascaded pipeline that combines speaker diarization with speech-LLM-based ASR, and a unified speech LLM that directly generates speaker labels, timestamps, and transcriptions. Our final submission is based on the cascaded pipeline, consisting of a fine-tuned DiariZen diarization model, a fine-tuned Qwen3-ASR model, and LLM-based generative error correction. For comparison, we also fine-tune VibeVoice-ASR as a unified model using the same official training data. All task-specific fine-tuning and model selection are performed using only the official MLC-SLM data, without external data or pseudo-labels. Experimental results demonstrate that the cascaded system remains more reliable under the MLC-SLM Task 1 conditions, while unified speech LLMs offer a promising direction for future speaker-attributed ASR.