Pupil and gaze signals predict lipreading ability better than mouth focus
Predicting Individual Differences in Lipreading From Eye-Tracking and Pupillometry During Silent Visual Speech
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
People differ a lot in how well they can understand speech by watching lips. This paper shows that how pupils react and subtle eye movements reveal more about lipreading skill than just looking at the mouth more. The researchers collected eye and pupil data while people silently lipread and used a neural network to find patterns linked to better or worse lipreading. This suggests new ways to measure lipreading ability objectively beyond just tracking where someone looks.
What this means in practice
- •For hearing aid designers: Use pupil and gaze signals to better assess and personalize visual speech perception support in hearing aids.
- •For user experience teams: Improve silent speech interfaces by integrating pupil and gaze metrics for real-time lipreading skill estimation.
Authors
Puneet Kumar, Satyarth Malik, Gao Zheng, Guoying Zhao, Xiaobai Li
Abstract
Lipreading ability varies widely, including among people who are hard of hearing. Although mouth-directed attention is linked to speechreading outcomes, it remains unclear how much lipreading skill can be inferred from viewing behavior and which signals carry this information. We examine whether individual differences in lipreading ability are reflected in viewing behavior during silent speech and quantify the contributions of gaze dynamics, pupil-linked signals, and spatial viewing allocation. We introduce the Lipreading Perception Study (LiPS) dataset and a tracker-independent preprocessing pipeline that synchronizes SR Research EyeLink gaze and pupil recordings with the stimuli. Twenty-four adults (age 32 $\pm$ 10 years) completed a four-alternative forced-choice silent lipreading task. An extreme-groups protocol yielded 18 participants labeled as stronger or weaker lipreaders. Each participant-trial included four aligned views: lip-region images, frame-level gaze, pupil summaries, and spatial viewing proportions. A four-branch neural network with cross-modal attention predicted labels using participant-disjoint splits and systematic ablations. The full model reached accuracy 0.680 $\pm$ 0.040 across five folds, outperforming naive fusion baselines in accuracy. The pupil and viewing-summary branch was most informative: removing it caused the largest degradation (0.618 accuracy, 0.623 AUC), while it alone achieved 0.675 accuracy. The lip-image stream contributed little to participant-level discrimination. Individual differences in lipreading are reflected in pupil-linked and frame-level gaze signals rather than in how much time a viewer spends looking at the mouth. Synchronized gaze and pupillary measures may provide objective markers of visual speech perception, warranting investigation in larger, clinically diverse populations. LiPS provides a resource for extending this work.