Speaker diarization model improves accuracy by avoiding over-tightening
Over-Tightening-Aware Pseudo-Labeling for Tight-Boundary Speaker Diarization
Sound
Summary
Speaker diarization means identifying who is speaking and when in an audio recording. Labels for training models often have extra padding or filled pauses, which makes the models guess loose boundaries. The authors studied why trying to tighten these boundaries too much can cause missed detections and errors. They proposed three fixes to avoid these problems and showed their method helps diarization and speech recognition work better.
What this means in practice
- •For speech technology teams: Improve speaker diarization boundary accuracy in multi-speaker speech recognition systems.
- •For call center software developers: Enhance diarization quality to better separate speakers and improve transcription accuracy in customer calls.
Authors
Shota Horiguchi, Takanori Ashihara, Marc Delcroix, Naohiro Tawara, Alexis Plaquet
Abstract
Training speaker diarization models on loose labels, such as speech segments with padded boundaries or filled pauses, often results in similarly loose model outputs. To obtain tighter boundaries, pseudo-labeling based on the averaged outputs of causal and anticausal models has been proposed. However, since the pseudo-labels are estimation-based, they can suffer from over-tightening, which increases missed detections that can propagate as unrecoverable errors to downstream tasks. This paper carefully analyzes the causes of over-tightening and proposes three approaches to address them: (i) removing pause filling rather than padding, (ii) introducing a burn-in phase to mitigate missed detections near the beginning of causal and anticausal predictions, and (iii) making pseudo-label-based co-training aware of the non-causal model used for final inference. Experimental results show that the proposed method reduces missed detections caused by over-tightening and improves both diarization accuracy and downstream multi-talker ASR performance.