Audio visual system improves dialogue clarity in noisy speech
Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
SoundArtificial IntelligenceHuman-Computer Interaction
Summary
Spoken dialogue systems that listen and talk at the same time often get confused when there is background noise or multiple people talking. To fix this, the authors created a special tool called AV-STE that uses both sound and lip reading from video to clean up what is being said before sending it to the conversation brain. This way, the main dialogue system stays the same and still understands and talks well. Their method made the conversation clearer and easier to follow when people were talking over each other or in noisy places.
What this means in practice
- •For voice assistant developers: Build more accurate voice assistants that handle noisy or overlapping speech by adding a modular visual speech cleaner before the main dialogue model.
- •For call center software teams: Improve spoken dialogue coherence in noisy customer service calls by using audio-visual token enhancement without retraining the core dialogue system.$Commercial implications: Enables commercial customer support platforms to deploy noise-robust conversational agents that maintain conversation flow in real-world noisy calls.
Authors
Bella Godiva, Yeonju Kim, Yong Man Ro
Abstract
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.