Target speech extraction works better with music in the background

MuseFlow-TSE: Music-Aware Target Speech Extraction via Cross-Source Band-Splitting and Data-to-Data Flow Matching

Sound

Summary

Sometimes when people talk with music playing, it’s hard to pick out just one person's voice. This paper introduces a method that can separate a target speaker’s voice from other sounds including music, other people talking, and noise. The method works in two stages: first, it separates different sound sources including music, and then it improves the speech clarity using a refined process. The authors show their approach helps keep the speaker’s identity clear and makes the extraction more accurate even with music present.

What this means in practice

Authors

Lu Ma, Yongliang Wang, Zhewen Su, Xin Tian

Abstract

Target speech extraction (TSE) isolates a speaker's voice given an enrollment utterance, but existing work is validated almost exclusively without music. Discriminative methods are content-faithful yet masking-bounded; generative methods improve naturalness but may hallucinate, drift in identity, and require many steps. We present MuseFlow-TSE, a discriminative deletion, generative insertion framework for music-bearing scenes. Stage-1 is a music-aware cross-source band-split extractor that estimates target, interfering speakers, music, and noise under conservation and music-leakage constraints. Stage-2 is a data-to-data flow refiner initialized by the discriminative estimate, negatively conditioned on interference stems, and run for 1--4 difficulty-adaptive Euler steps with an end-to-end fidelity objective.