Driver gaze prediction improves during tracker loss using context and scene data
Context-Aware Causal Gaze Forecasting for Human-Vehicle Interaction During In-Cabin Tracking Dropouts
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Sometimes, devices that track where a driver is looking stop working when the driver turns their head sharply, like when checking mirrors. This is a problem because knowing where the driver is looking at those moments is important for safety. The authors created a way to predict the driver's gaze during these tracker dropouts by looking at recent head and gaze movements and what the driver might be seeing around them. Their approach uses real driving data and shows better accuracy when it updates with what the driver is seeing during the dropout compared to only using past information.
What this means in practice
- •For automotive safety engineers: Enhance in-cabin driver monitoring systems by accurately predicting gaze during moments when trackers temporarily lose sight, improving safety interventions.
- •For advanced driver assistance system developers: Integrate causal gaze forecasting to maintain eye-tracking functionality during occlusions or head turns, supporting better driver attention models.
Authors
Shabnam Shabani, Ghazal Farhani
Abstract
Dashboard-mounted gaze trackers often lose sight of the driver's eyes during large head rotations, including shoulder checks, mirror glances, and intersection scanning. These maneuvers occur when information about the driver's visual attention is most useful. Offline gap-filling methods may reconstruct a missing interval using observations from both sides, but an online driver-monitoring system cannot rely on measurements that have not yet occurred. We therefore formulate causal gaze recovery: forecasting unavailable gaze at time t without target-tracker gaze at t or later. We introduce the Causal Context-Gated Forecaster (CCGF), which encodes a 60-frame pre-dropout history of gaze and head pose and combines it with DINOv3 scene features. A learned reliability gate controls the contribution of the history and scene representations as the dropout progresses. We evaluate two scene conditions: Live, in which the scene representation continues to update during tracker loss, and Frozen, in which the final pre-dropout representation is used throughout the missing interval. We evaluate CCGF on 2,047 eligible, naturally occurring GazeSense head\_lost events drawn from 10.5 h of naturalistic driving by ten drivers. Across all recordings, head\_lost accounts for 8.5 percent of GazeSense recording time. Synchronized gaze coordinates from a head-mounted Neon tracker provide supervision and evaluation targets but are never used as model inputs. Under leave-one-driver-out evaluation, CCGF achieves a mean per-driver median error of 175.7 px (10.5 deg) with Live scene updates, a 33 percent reduction relative to history-only causal forecasting. With Frozen scene input, the error increases to 210.8 px (12.9 deg), indicating that scene observations acquired during the dropout provide useful predictive information. We will release the dataset, evaluation protocol, and causal baselines.