Video to speech system uses text to improve accuracy and naturalness
Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning
Computer Vision and Pattern RecognitionSound
Summary
Turning silent videos of people talking into speech is tricky because the mouth movements alone don’t always show exactly what’s being said. The authors propose a new method that adds the actual text of what should be spoken as an extra clue. This helps the model figure out the right words and sounds more accurately and naturally. Their experiments found that this approach makes the generated speech match the video closely while sounding almost human.
What this means in practice
- •For video editors: Create accurate voice tracks from silent talking-face videos with improved speech clarity and timing.
- •For assistive technology developers: Develop communication aids that produce clearer synthetic speech from visual lip movement inputs and textual cues.
Authors
Gunwoo Lee, Yoori Oh, Yoseob Han
Abstract
Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance. To address this, we propose Watch Your Speech (WYS), a video-to-speech synthesis framework that incorporates textual conditioning as an explicit linguistic cue to mitigate visual ambiguity. Our framework features an attention-based embedding fusion module that synergistically integrates textual context with video sequences, coupled with a conditional flow matching objective for high-fidelity speech generation. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that WYS achieves superior performance, establishing new state-of-the-art results in audio-visual synchronization (LSE-C/D) while maintaining highly competitive textual accuracy (WER). Subjective evaluations further confirm that our model generates speech with near-human naturalness, validating the effectiveness of textual conditioning in content-controlled video-to-speech synthesis. Project page: https://github.com/gunwoo5034/Watch-your-Speech