Vision centric system predicts pedestrian crossing actions from video frames
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification
Computer Vision and Pattern Recognition
Summary
Predicting what a pedestrian will do next just by looking at regular video frames is hard because the system needs to focus on the right person in every frame without extra sensors. The authors propose a new method called ViCross that uses advanced AI models handling both images and language to keep track of pedestrians efficiently. They solve problems like how to focus on important parts of the image without using too much computing power and how to understand motion over time. Their method works well compared to others that use lots of different data sources. They also made their code available for others to use.
pedestrian action predictionvision-centricmultimodal large language modelsvariable resolution patch mappingspatial constraint enhancementvideo frame analysistarget-centric reasoningtoken allocationspatiotemporal priors
Authors
Yao Tian, Le Yang, Binglu Wang
Abstract
Vision cues are available and informative for pedestrian action prediction, but obtaining stable target-centric representations from video frames remains challenging without frame-level external perception cues. Thus, most methods rely on additional perception modules or multi-source information fusion, leaving the reliability of vision-centric setting an open question. To this end, we propose ViCross, a vision-centric pedestrian crossing action prediction framework powered by multimodal large language models, which maintains target-centric reasoning from video frames without additional perception modules beyond first-frame target initialization. While multimodal large language models exhibit strong visual understanding, applying them directly to vision-centric action prediction faces two challenges. First, accurately perceiving target pedestrians often requires high resolution inputs and dense visual tokenization, making full-frame encoding computationally prohibitive. ViCross tackles this with Variable Resolution Patch Mapping module for efficient token allocation while preserving key pedestrian details. Second, missing spatiotemporal priors hinder consistent cross frame reasoning. ViCross mitigates this with a Spatial Constraint Enhancement Strategy that captures past motion, future locations, and action semantics for training-time proactive spatial rectification. Extensive experiments show that ViCross delivers clear gains in vision-centric prediction settings and is competitive with multi-source fusion approaches in several settings. Code is available at https://github.com/2tianyao1/ViCross.git.