Falcon cuts streaming speaker extraction size and latency drastically
Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Sound
Summary
Listening to one person speak clearly in a noisy, crowded place is hard for computers, especially when they have to do it quickly and with limited computing power. The authors introduce a new set of realistic tests called RealSSA that better simulate real noisy environments with many speakers. They also create Falcon, a new lightweight method that separates a target speaker’s voice from others more efficiently and faster than before. This method works well even on small devices and in tough situations where people speak at different distances or sudden noises appear.
What this means in practice
- •For voice assistant developers: Run real-time speaker voice extraction reliably on mobile or smart home devices with limited computing power and complex noise.$Commercial implications: Enables consumer devices to deliver clearer voice recognition in noisy environments by efficiently isolating target speakers.
- •For video conferencing platform engineers: Improve live meeting audio by extracting individual speakers' voices quickly, even with multiple overlapping talkers and background noise.
Authors
Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu
Abstract
Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.