Self supervised speech model improves multi speaker separation and diarization
SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Computation and LanguageMachine Learning
Summary
Understanding multiple people talking at once is hard for many speech recognition systems because they are usually trained on single speakers. The authors created SepRQ, a new system that learns to separate voices without needing masked audio parts for training. SepRQ works well in identifying who spoke when and separating voices, even for three speakers at the same time, which is usually very challenging. It is also open-source, allowing others to use and build on it.
What this means in practice
- •For speech technology developers: Improve speech systems to separate and recognize multiple speakers simultaneously, enhancing diarization and recognition accuracy in complex audio.
- •For conference transcription teams: Generate more accurate transcriptions by separating overlapping voices in meetings and multi-speaker settings using SepRQ’s open-source model.
Authors
Séverin Baroudi, Hervé Bredin, Ricard Marxer
Abstract
Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-speakers scenarios. We present SepRQ, an open-source SSL framework that replaces masked prediction with a pseudo-source-separation objective over frozen random-projection codebooks. By adopting a novel mask-free, multiresolution approach, SepRQ achieves state-of-the-art performance in Speaker Diarization and Speech Separation on the SUPERB benchmark, surpassing WavLM and other cocktail-party derived SSLs at both Base and Large scales, while requiring only 85.68M inference parameters. SepRQ also demonstrates strong performance across target-speaker tasks requiring enrollment (such as Target-Speaker Automatic Speech Recognition), and on the challenging multi-domain DIHARD 3 diarization dataset. Notably, we report strong separation capabilities on three-speaker mixtures (WSJ0-3Mix), where current SSL literature struggles. While cocktail-party SSLs remain scarce and closed-source, limited to C-HuBERT and the enrollment-based SA-WavLM, we open-source SepRQ to the community.