Human gaze improves visual reasoning in AI models

Learning Grounded Reasoning from Human Gaze

Computer Vision and Pattern Recognition

Summary

Humans look around in specific ways when solving visual problems, and where they look can guide AI models to focus better. The authors collected lots of human gaze data during tricky visual tasks and found that selectively using gaze data helps AI reason more like humans. They developed a new system called GGR that uses this guided gaze information to improve accuracy and reasoning behavior. Interestingly, teaching the AI by reward rather than copying gaze directly worked well too. This shows that human gaze helps guide where AI looks, without copying every detail of human behavior.

What this means in practice

  • For computer vision developers: Improve AI systems that analyze complex visual scenes by integrating selective human gaze data to enhance visual grounding and reasoning accuracy.
  • For human-computer interaction teams: Design gaze-guided interfaces that improve AI assistance by aligning machine attention with human visual exploration patterns.

Authors

Gabriel Sarch, Declan Campbell, Nicholas Budny, Kia Ghods, Katerina Fragkiadaki, Jonathan D. Cohen, Brenden M. Lake, Taylor W. Webb

Abstract

Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,109 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that using human gaze as a prior for grounded reasoning consistently outperforms supervision using the full gaze sequence. Based on these findings, we introduce GGR (Gaze-Guided Reasoner), which achieves greater human alignment and more reliable visual grounding while matching or outperforming leading grounded-reasoning baselines in accuracy. Reinforcement learning further improves task accuracy, gaze alignment, and grounding quality despite using no gaze supervision. Behavioral analyses show that gaze supervised models exhibit more visual exploration before arriving at an answer. Our results suggest that the value of human gaze lies in guiding spatial exploration and shaping reasoning behavior without requiring complete imitation of human behavior.