Single AI model improves speech extraction from audio and video cues
Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues
SoundMultimedia
Summary
Separating one person's speech when many talk at once can be tricky. The authors showed that one AI model can use different kinds of clues—like lip movements, gestures, voice samples, or transcribed words—to pick out the target speech. The model predicts speech step by step, remembering what it has already identified, so it keeps working even if some visual information is missing. It performs as well as other top methods and can work in real-time settings.
What this means in practice
- •For video conferencing platforms: Enhance voice clarity by extracting target speech using both visual and audio cues even when video frames drop.$Commercial implications: Enables more reliable speech separation products for conferencing providers by combining audio and visual information in one model.
- •For hearing aid developers: Integrate a single model that uses real-time audio-visual cues to improve speech focus amid background noise and multiple speakers.$Commercial implications: Supports development of advanced hearing aids that better isolate target speech in complex environments with fewer models.
Authors
Wenxuan Wu, Shuhan Zhang, Shuai Wang, Haizhou Li
Abstract
Target speech extraction (TSE) typically trains a separate extractor per cue, and visual-cue systems often need corruption-matched training to remain robust under visual frame corruption. We show that one autoregressive LLM backbone, TSE-Omni, can serve both temporally synchronous cues (lip movements, co-speech gestures) and asynchronous cues (enrollment audio, text). TSE-Omni is driven by next-token prediction: each step predicts target speech semantic tokens from its own past outputs, which we term self-enrollment, forming a continuous target-speech context initialized by the enrollment cue (asynchronous audio or text, or a short visual prefix). This enables audio-visual compensation: the model uses synchronized visuals when intact and its token history when visual frames are missing. Under clean visuals, TSE-Omni matches strong discriminative and generative baselines (SpeechBERTScore 0.81 on VoxCeleb2 and 0.89 on LRS3 zero-shot) with higher DNSMOS. On the same VoxCeleb2 test set, after a 2 s clean visual start, removing the remaining visual frames leaves SpeechBERTScore at 0.81. It remains usable under sparse overlap and multi-speaker interference, and supports streaming inference. Project page: https://alexwxwu.github.io/tseomni-main/.