Improving audio source separation with better data and control signals
A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine
Sound
Summary
Separating sounds from a mixed audio recording is hard, especially when trying to pick out specific types of sounds described in natural language. The authors developed a system that automatically picks better training data and improves the way the model uses language to identify sounds. Their tests show this approach works better across different sound types. People who listened also preferred the improved results over current popular systems.
What this means in practice
- •For audio software developers: Create apps that isolate specific sounds from recordings more accurately using optimized language queries and curated training data.
- •For media production teams: Improve post-production workflows by separating desired sounds from complex audio sources with better accuracy and natural language commands.
Authors
Amirhossein Hajavi, Hanhee Lee, Pushya Jain, Sky Qiao, Emmanuel Ko, Yuanhao Yu, Irina Kezele
Abstract
Language-queried audio source separation (LASS) enables extracting any sound source using natural language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.