Improving speech recognition with synthetic lip movement videos
Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation
Computation and LanguageSound
Summary
Speech recognition systems that also look at lip movements can understand speech better. But it’s hard to get many videos that show people talking with matching audio and visual data. The authors created realistic videos of lip movements using audio recordings alone to help train these systems. They found that adding these fake videos to real ones makes the system more accurate and that even using just the fake videos can work well for languages without real video data.
What this means in practice
- •For speech recognition developers: Improve audiovisual speech recognition accuracy by augmenting training datasets with synthetic lip-synced videos from audio-only data.
- •For language technology teams: Create baseline audiovisual speech models for languages lacking visual speech data using synthetic talking-head video generation.
Authors
Pol Buitrago, Pol Gàlvez, Javier Hernando
Abstract
Audiovisual Speech Recognition (AVSR) is a multimodal approach to speech recognition that incorporates visual information from lip movements to enhance model performance. Despite its advantages, its development remains constrained by the limited availability of labeled audiovisual (AV) datasets. This work explores the use of synthetic visual data as a solution, using an audio-driven talking-head pipeline to generate lip-synchronized visual content from existing audio data. We evaluate the effectiveness of synthetic visual data both as an augmentation strategy and as a standalone training resource, applying our approach to Spanish and Catalan. Our results show that augmenting real AV data with synthetic samples yields relative Word Error Rate (WER) reductions of up to 16.2%, demonstrating the potential of this approach. Moreover, we demonstrate that synthetic data alone can serve as a baseline for AVSR training in languages lacking AV datasets. These findings provide evidence that synthetic visual data can serve as a scalable solution to AVSR data scarcity, enabling broader language coverage.