Visual speech understanding improves with face and lip modeling
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
Computer Vision and Pattern Recognition
Summary
Lipreading from videos is hard because facial movements alone don’t always clearly show speech sounds, especially with different head positions or low video quality. The authors created FLARE, a system that learns to lipread better by looking at both the lips and the whole face at the same time. It also uses speech sounds to teach the visual model during training, making it better at understanding silent video speech. FLARE performs well in tests on multiple public lipreading datasets and handles difficult video conditions more robustly.
What this means in practice
- •For speech technology developers: Build improved silent speech recognition tools that work well despite video quality or pose changes by integrating FLARE’s joint face-lip modeling approach.$Commercial implications: Enables creation of products that decode speech from video alone, useful for privacy-focused or noisy environments.
- •For security system engineers: Enhance lipreading-based authentication or monitoring by using FLARE’s robust visual speech features resilient to head movement and video degradation.
Authors
Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Jia Pan, Yumei Zhang, Lichen Zhang
Abstract
Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at https://github.com/jxzhanggg/FLARE.