Multi camera system estimates gaze target location in three dimensions
Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching
Computer Vision and Pattern Recognition
Summary
Estimating where someone is looking in 3D space is important but challenging, especially without detailed labeled data. The researchers developed a method called Self-MVGTE that uses images from multiple cameras and combines imperfect 2D gaze guesses into a more accurate 3D gaze target. They handle uncertainty in the data by modeling gaze directions as cones of possible targets and use features from additional models to guide their predictions. Their approach works well without needing manual labels and outperforms previous methods that required full supervision.
3D gaze estimationmulti-view camerasself-supervised learningpseudo labelsprobabilistic ray marchinggaze conesdepth-guided feature samplingDINOv2Depth-Anything-3MVGT dataset
Authors
Keqi Chen, Vinkle Srivastav, Nicolas Padoy
Abstract
We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views. Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time. Moreover, it does not require any ground-truth annotations from the target scene and uses only the multi-view input images from a calibrated camera setup, pseudo 2D gaze target labels from a monocular gaze target estimation model, and 3D gaze vectors from a monocular 3D gaze estimation model. A key challenge is that these pseudo labels are inherently noisy and multi-view inconsistent. To address this, we propose a probabilistic ray marching framework, which models the uncertainty of these pseudo labels and exploits 3D gaze vectors as geometric priors. Specifically, these gaze vectors are first integrated into the monocular gaze target estimation model to improve its generalization to unseen scenes, producing higher-quality pseudo labels. Then, for 3D gaze target estimation, we construct a 3D gaze cone by casting a bundle of rays from the eye position around the gaze vector to strictly constrain the solution space. Within this cone, we propose a depth-guided feature sampling strategy using off-the-shelf DINOv2 and Depth-Anything-3 models, and estimate a spatial likelihood distribution of the gaze target. Finally, we convert the pseudo gaze target labels into a target distribution and softly optimize the network. Extensive experiments on the MVGT dataset show that Self-MVGTE achieves state-of-the-art performance, surpassing existing fully-supervised baselines.