Audio visual system improves dialogue in noisy speech interactions
Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
SoundArtificial IntelligenceHuman-Computer Interaction
Summary
People talking with voice assistants often face problems when there's background noise or people talking over each other, causing the assistant to misunderstand. The authors designed a new tool called AV-STE that uses both sound and lip-reading video to clean up the speech input before it reaches the main conversation AI. This means the AI doesn't have to be retrained to handle video inputs, saving time and effort. With AV-STE, the conversation AI gave more coherent answers even when the audio was unclear. The improvement also worked well when tested with different types of dialogue systems.
full-duplex spoken dialogueaudio-visual speech processingspeech tokensbackground noiseoverlapping speechspeech LLMmultimodal trainingfrozen modelsemantic restoration
Authors
Bella Godiva, Yeonju Kim, Yong Man Ro
Abstract
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.