Audio-visual speech separation improves clarity in noisy environments
CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge
SoundArtificial Intelligence
Summary
Separating a single person's speech from noisy surroundings is hard, especially when other people talk at the same time. The authors designed a two-step method that first roughly separates the target speaker's voice using video of their lip movements, then refines the audio further by combining multiple estimates. Their approach won top results in a real-world speech separation challenge, showing it can clearly recover speech even with noise and overlapping speakers.
What this means in practice
- •For telecommunications engineers: Improve voice call clarity by isolating the target speaker's voice in noisy, reverberant settings using synchronized video input.
- •For assistive technology developers: Enhance hearing aids or communication devices by refining speech separation using visual cues of the speaker's face in real-world noisy environments.
Authors
Longjie Luo, Zhongshu Hou, Yuepeng Li, Gang Yang, Shidong Shang, Fan Zhou
Abstract
Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage AVSE framework. In the first stage, a progressive SPAV-TFGridNet is extended to two synchronized visual streams to produce visually ordered coarse estimates by progressively suppressing interfering speech while preserving target reverberation and shared background noise. In the second stage, a TFGridNet-based target speaker refinement network takes the original mixture and both coarse estimates as input to further recover target speech and improve signal quality. The coarse and refined estimates are finally combined using development-set-calibrated fusion. On the ISCSLP 2026 Real-World AVSE Challenge, our system tied for first place on Track 1 and ranked second on Track 2 in OVRL, while achieving the lowest CER on both tracks at 12.31% and 14.79%, respectively.