Speech language models improve separating multiple talkers in noisy audio
SLM-SS: Token-Based Speech Separation with Speech Language Models
SoundArtificial Intelligence
Summary
Understanding a mix of several people talking at once is hard for computers. The authors created a new method that breaks down speech into tokens — like word pieces — and uses a special language model designed for speech to separate each speaker’s voice. This approach helps keep the meaning clearer and makes the separated speech easier to understand, even with background noise or on languages the model wasn’t trained on. Their method showed better understanding scores and sound quality compared to previous systems.
What this means in practice
- •For voice assistant developers: Improve voice assistants’ ability to separate and understand multiple speakers talking simultaneously in real-world noisy environments.
- •For call center technology teams: Enhance transcription accuracy for overlapping speech in customer service calls by using token-based speech separation models.
Authors
Chenda Li, Tianhua Li, Yiming Chen, Wei Wang, Jiahe Wang, Xin Zhou, Yanmin Qian
Abstract
Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves $6.92\%$ WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.