Pose adaptive feature modulation improves lip reading accuracy
Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition
Computer Vision and Pattern Recognition
Summary
Visual speech recognition, or lip reading by computers, can struggle when the speaker's head pose changes and affects how their lips look. The authors found that simply mixing multiple fixed adjustments for different poses can make recognition worse. They created a new method that dynamically adjusts the strength of these pose-related changes based on the input, improving the accuracy of recognizing spoken sounds from video. Their method especially favors deeper adjustments when the head pose varies more, leading to better performance on standard lip reading datasets.
What this means in practice
- •For speech technology developers: Improve lip reading models for devices that need to understand speech visually despite head movements.
- •For human computer interaction teams: Enhance visual speech interfaces to better handle changes in user head pose for reliable communication aids.
Authors
Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
Abstract
Head-pose variation introduces substantial appearance transformations in visual speech recognition (VSR), making pose-aware feature modulation desirable. However, performance degradation and unwanted feature interactions may result from using numerous Feature-wise Linear Modulation (FiLM) circuits with fixed modulation intensity. We propose a Pose Adaptive Dynamic FiLM framework with a Dynamic Residual FiLM (DR-FiLM) modulator that predicts input-dependent weights to adaptively control the strength of pose-conditioned modulation. Experiments on LRS2 and LRS3 demonstrate that unweighted multi-pathway modulation substantially degrades phoneme recognition, increasing PER to 20.33% and 29.42%, respectively, compared with 16.20% and 20.96% for the single ResFiLM configuration. In contrast, the proposed DR-FiLM with dynamic Deep-Res weighting reduces PER to 15.74% on LRS2 and 23.91% on LRS3, substantially mitigating the adverse effects of unweighted modulation. The analysis of the learned weights further reveals a consistent tendency to assign greater weight to the deeper FiLM pathway as head-pose variation increases. These results show that merging pose-conditioned FiLM circuits is more efficient when the modulation strength is dynamically controlled.