Video and clicks improve sound separation for similar on-screen sources
SelectSep: Target-Selective Video Queries for Audio Source Separation
MultimediaSound
Summary
Separating sounds from a video with multiple people or objects making noise is tricky when they look alike. The authors propose SelectSep, which uses a short text description plus a single click on the video to pick out the sound of just one target object. This approach helps the computer focus better on the intended source without needing detailed segmentation of the image. They tested it on examples where both the target and interfering sounds are visible and similar, showing improved sound separation results.
What this means in practice
- •For audio software developers: Create tools that separate specific sounds from videos with multiple similar-sounding objects using simple user inputs like text and clicks.
- •For video editors: Isolate particular sounds in complex scenes for cleaner audio editing without manual segmentation or detailed labeling.
Authors
Suyeon Lee, Doyeop Kwak, Joon Son Chung
Abstract
Video-queried sound separation (VQSS) aims to isolate a target sound from an audio mixture using its corresponding video as a query. However, scene-level video queries can become ambiguous when multiple sounding objects are visible, particularly when the target and interferer are semantically similar. We propose SelectSep, a target-selective VQSS framework that jointly uses text and a single click coordinate to specify the target source and instance without requiring segmentation at inference time. We adapt the visual encoder through target-feature distillation with text and click conditioning, and then fine-tune the separator using the adapted visual features. We also introduce an on-screen interference setting, where both the target and interferer are visible, to evaluate target selectivity. Experiments on standard benchmarks and the on-screen interference setting demonstrate consistent improvements in separation quality and target selectivity, particularly for mixtures containing semantically similar sources.