Text-guided model extracts and enhances speakers from audio mixtures
Textraction: Unified Text-Guided Speaker Extraction and Enhancement for Single- and Multi-Channel Audio
Sound
Summary
Mixes of sound can include multiple people talking or background noises, making it hard to focus on one voice. The authors created Textraction, a single model that uses natural language prompts like descriptions or spatial info to pick out specific speakers or noises from audio. It works with both simple and complex recordings and can either keep or remove certain sounds based on instructions. Textraction avoids complex steps other methods need by directly extracting the target voice or noise in one go. The authors also designed a small text encoder that understands instructions well without needing large, costly language models.
What this means in practice
- •For audio engineers: Improve audio editing by selectively isolating or removing speakers or noise from recordings using natural language instructions.
- •For voice assistant developers: Enhance speech recognition by isolating target speakers in noisy environments based on textual or spatial cues without complex preprocessing.
Authors
Aviad Eisenberg, Sharon Gannot, Shlomo E. Chazan
Abstract
We present Textraction, a text-guided speaker extraction framework that enables the selective extraction of desired speakers or noise sources from an audio mixture using only natural-language descriptions. The proposed method operates within a single unified model that supports both single-channel and multi-channel inputs and it allows conditioning on either semantic textual descriptions of the target source or on its spatial description. Dedicated single- and multi-channel encoders feed a shared stack of time--frequency transformer blocks, which is conditioned on the prompt and on the recording configuration in every block through cross-attention and estimates the target directly from the mixture in one forward pass, without a separation stage or a source-selection stage. The system can extract an arbitrary subset of entities from a mixture, assuming up to two simultaneously active speakers and two noise sources, and interprets prompts specifying which entities to keep, which to remove, or both. In addition, we show that the framework naturally applies to a speech-enhancement setting, in which a single target speaker is present alongside background noise. To circumvent the application of computationally costly large language models used in prior work, we introduce a lightweight, task-adapted text encoder obtained by contrastively fine-tuning a compact text-retrieval model, which learns to be sensitive to instruction polarity (e.g., keep vs. remove) while remaining robust to paraphrasing. Experiments across attribute-based extraction, spatially guided extraction, and speech enhancement show that the proposed approach effectively isolates the desired entities under various acoustic and spatial conditions, and matches or surpasses dedicated spatial-extraction and enhancement baselines while offering substantially greater flexibility.