Self supervised approach estimates 3d gaze targets from multiple cameras
Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching
Computer Vision and Pattern Recognition
Summary
Accurately finding where someone is looking in 3D space is hard and usually needs lots of manual data. The authors created a method that uses multiple camera views and existing rough gaze estimates to figure out the 3D point of gaze without needing exact labels. They deal with noisy input by modeling uncertainty and geometric clues, improving the accuracy of gaze targets in 3D scenes. Their approach performed better than earlier methods that relied on full supervision.
What this means in practice
- •For virtual reality developers: Build virtual reality systems that track users’ 3D gaze targets more accurately using multi-camera setups without manual annotations.
- •For automotive engineers: Improve driver attention monitoring by estimating where drivers are looking in 3D space with multiple in-car cameras without requiring extensive labeled data.
Authors
Keqi Chen, Vinkle Srivastav, Nicolas Padoy
Abstract
We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views. Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time. Moreover, it does not require any ground-truth annotations from the target scene and uses only the multi-view input images from a calibrated camera setup, pseudo 2D gaze target labels from a monocular gaze target estimation model, and 3D gaze vectors from a monocular 3D gaze estimation model. A key challenge is that these pseudo labels are inherently noisy and multi-view inconsistent. To address this, we propose a probabilistic ray marching framework, which models the uncertainty of these pseudo labels and exploits 3D gaze vectors as geometric priors. Specifically, these gaze vectors are first integrated into the monocular gaze target estimation model to improve its generalization to unseen scenes, producing higher-quality pseudo labels. Then, for 3D gaze target estimation, we construct a 3D gaze cone by casting a bundle of rays from the eye position around the gaze vector to strictly constrain the solution space. Within this cone, we propose a depth-guided feature sampling strategy using off-the-shelf DINOv2 and Depth-Anything-3 models, and estimate a spatial likelihood distribution of the gaze target. Finally, we convert the pseudo gaze target labels into a target distribution and softly optimize the network. Extensive experiments on the MVGT dataset show that Self-MVGTE achieves state-of-the-art performance, surpassing existing fully-supervised baselines.