Soloist separates voices from noisy and overlapping speech
Soloist: A Generative Model for Blind Voice Separation in the Wild
Sound
Summary
Separating individual voices from recordings with many people talking over each other, background noise, and music is very hard, especially when you don’t know who or how many people are speaking. The authors introduce Soloist, a smart audio model that listens to mixed voices and recreates each speaker’s voice separately without needing prior information on the number of speakers. They teach Soloist in two phases using both made-up and real audio, improving its ability on real-world sounds. Soloist works well on conversations, noisy environments, and even music with singing, making it useful for processing real conversations accurately.
What this means in practice
- •For call center operators: Improve transcription accuracy by separating overlapping voices in noisy customer calls for clearer records and responses.
- •For music producers: Extract singing vocals from music tracks with accompaniment while preserving vocal pitch and clarity for remixing or editing.
Authors
Chen Yang, ChuFan Yu, Botian Jiang, Jingqi Chen, Wenxuan Wang, Yitian Gong, Zhaoye Fei, Xipeng Qiu
Abstract
Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.