Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models
2026-06-08 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors studied how to predict when pedestrians are about to cross the street using short videos taken from a first-person view, like a driver's perspective. They treated this problem like answering a yes/no question about what the pedestrian intends to do and used advanced vision-language models to make predictions. Initially, these models did only a little better than random guessing, but after fine-tuning them specifically for this task, the models improved a lot. Adding extra helpful information, like the motion of the vehicle and where the driver is looking, made the predictions even better. Their best model beat existing methods by a noticeable margin in accuracy.
egocentric visionpedestrian intent predictionvisual question answering (VQA)vision-language models (VLMs)zero-shot learningparameter-efficient fine-tuningego motioneye gaze trackingtransformer modelstraffic safety
Authors
Danya Li, Xiang Su, Yan Feng, Rico Krueger
Abstract
Egocentric vision offers a first-person view of human perception and decision making, yet its potential for traffic-safety prediction remains underexplored. In this work, we study the decoding of pedestrian crossing intentions from short egocentric video clips. We approach this by formulating the task as a closed-ended visual question answering (VQA) problem and leveraging vision language models (VLMs) to predict the pedestrians' intent. We first benchmark three families of state-of-the-art VLMs in a zero-shot setting, finding that they achieve moderate gains over random guessing but exhibit limited higher-level traffic reasoning. Motivated by these findings, we further adapt VLMs to the target task using parameter-efficient fine-tuning. Our results show that the fine-tuned models substantially outperform their zero-shot counterparts and achieve a 9\% accuracy improvement over a specialized transformer-based baseline. Finally, we demonstrate that incorporating additional contextual cues, including ego motion, vehicle motion, and eye gaze, further improves predictive performance. In particular, the fine-tuned Qwen3-VL-2B model guided by eye gaze and ego motion achieves a 14.5% accuracy improvement over the transformer baseline, establishing a new state of the art for egocentric pedestrian intent decoding.