Target speech extraction works better with music in the background
MuseFlow-TSE: Music-Aware Target Speech Extraction via Cross-Source Band-Splitting and Data-to-Data Flow Matching
Sound
Summary
Sometimes when people talk with music playing, it’s hard to pick out just one person's voice. This paper introduces a method that can separate a target speaker’s voice from other sounds including music, other people talking, and noise. The method works in two stages: first, it separates different sound sources including music, and then it improves the speech clarity using a refined process. The authors show their approach helps keep the speaker’s identity clear and makes the extraction more accurate even with music present.
What this means in practice
- •For audiovisual content editors: Isolate a single speaker’s voice from recordings with music and other speakers for clearer audio editing.
- •For call center operators: Enhance customer speech clarity in noisy call recordings that include background music or multiple talkers.
Authors
Lu Ma, Yongliang Wang, Zhewen Su, Xin Tian
Abstract
Target speech extraction (TSE) isolates a speaker's voice given an enrollment utterance, but existing work is validated almost exclusively without music. Discriminative methods are content-faithful yet masking-bounded; generative methods improve naturalness but may hallucinate, drift in identity, and require many steps. We present MuseFlow-TSE, a discriminative deletion, generative insertion framework for music-bearing scenes. Stage-1 is a music-aware cross-source band-split extractor that estimates target, interfering speakers, music, and noise under conservation and music-leakage constraints. Stage-2 is a data-to-data flow refiner initialized by the discriminative estimate, negatively conditioned on interference stems, and run for 1--4 difficulty-adaptive Euler steps with an end-to-end fidelity objective.