System locates multiple sound sources from natural language queries

QC-MI-OVSEL: Benchmarking Query-Conditioned Multi-Instance Open-Vocabulary Sound Event Localization

Sound

Summary

This paper deals with the challenge of finding all sounds in a 3D space that match a spoken or written category, even if that sound isn't present. The authors introduce a new large dataset that tests this by including many scenes, sound categories, and cases where sounds may not appear at all. They present a system called Q-RSet that uses information from natural language queries to detect and locate multiple sound sources at once. Their tests show that this method improves how well sound sources are found, positioned, and counted compared to previous models. The system also handles new, unseen sound categories and complicated queries reasonably well.

What this means in practice

  • For audio engineers: Identify and locate multiple instruments or sounds in spatial audio recordings from natural language descriptions to aid mixing or sound design.
  • For robotics developers: Equip robots with the ability to recognize and localize multiple sound sources described by users, improving situational awareness in dynamic environments.

Authors

Yongqiang Chen, Peiwei Chang, Yuelan Cheng, Xiatian Zhu, Wenwu Wang

Abstract

We study query-conditioned multi-instance open-vocabulary sound event localization (QC-MI-OVSEL), where a system must recover all active physical sources matching a natural-language category query, including when the queried category is absent. We introduce the QC-MI-OVSEL benchmark, a 40,000-scene first-order Ambisonics benchmark with 80,000 balanced present/absent query views over 83 classes, including 17 novel classes excluded from training and development. Each frame is annotated with the activity and 3-D direction of every query-matching source, with up to three simultaneous targets. We further propose Q-RSet, which uses query-conditioned source slots and responsibility-based evidence aggregation for source-set prediction. Experiments show improved source recovery, localization recall, cardinality estimation, exact-set recovery, and absent-query suppression over complementary baselines. Analysis indicates that novel categories, paraphrased queries, and higher source multiplicity mainly impair source discovery and set completion, while directional accuracy remains comparatively stable for successfully recovered sources.
View PDFOpen arXiv