Egocentric gaze prediction improves with flow based trajectory modeling

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

Computer Vision and Pattern RecognitionMachine Learning

Summary

Predicting where people look when moving around is tricky because their eye movements change quickly and depend on both what they see and what they are trying to do. The authors created GazeFlow, a method that guesses not just single gaze points but whole gaze paths by combining visual details and task goals. It works by starting with random data and transforming it step-by-step into realistic eye movement patterns, matching how people actually look at things over time. Tested on common benchmarks, GazeFlow predicts eye positions more accurately than earlier methods.

What this means in practice

  • For augmented reality developers: Create more natural and accurate gaze-driven interactions by predicting users’ eye movement paths during tasks in AR environments.
  • For video game designers: Improve non-player character attention and gaze behaviors by simulating realistic human gaze dynamics influenced by visual stimuli and goals.

Authors

Sheng Zhao, Weikai Lin, Yuhao Zhu

Abstract

Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.