Speaker extraction model improves separation using multiple prompts
PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment
Sound
Summary
Separating one person's voice from a noisy recording with many people talking is hard. The authors created a new system that listens better by using different clues, like a person's voice sample, their speech transcript, or descriptions about them. Their model generates separated speech more clearly than older methods. It also works well in different noisy situations by combining these clues smartly.
What this means in practice
- •For voice assistant developers: Improve voice assistants' ability to isolate a user's voice from noisy environments using multiple cues beyond audio samples.
- •For audio transcription services: Enhance transcription accuracy by separating individual speakers in multi-speaker recordings using generative speaker extraction with multimodal prompts.
Authors
Yutong Wen, Ke Chen, Prem Seetharaman, Haonan Chen, Oriol Nieto, Paris Smaragdis, Justin Salamon, Zeyu Jin
Abstract
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.