Vision centered method improves pedestrian crossing predictions
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification
Computer Vision and Pattern Recognition
Summary
Pedestrian crossing prediction helps machines guess when people will cross streets, improving safety. The authors designed ViCross, a system that focuses only on video frames to predict pedestrian actions without extra sensors. It smartly selects important image parts and uses past and future motion information to make better guesses. Their tests show ViCross works well compared to other methods that need more data sources.
What this means in practice
- •For autonomous vehicle developers: Enable prediction of pedestrian crossing actions using only visual input from vehicle cameras without relying on extra sensors.
- •For urban traffic monitoring teams: Improve pedestrian behavior prediction for better traffic flow and safety analysis using standard video surveillance footage alone.
Authors
Yao Tian, Le Yang, Binglu Wang
Abstract
Vision cues are available and informative for pedestrian action prediction, but obtaining stable target-centric representations from video frames remains challenging without frame-level external perception cues. Thus, most methods rely on additional perception modules or multi-source information fusion, leaving the reliability of vision-centric setting an open question. To this end, we propose ViCross, a vision-centric pedestrian crossing action prediction framework powered by multimodal large language models, which maintains target-centric reasoning from video frames without additional perception modules beyond first-frame target initialization. While multimodal large language models exhibit strong visual understanding, applying them directly to vision-centric action prediction faces two challenges. First, accurately perceiving target pedestrians often requires high resolution inputs and dense visual tokenization, making full-frame encoding computationally prohibitive. ViCross tackles this with Variable Resolution Patch Mapping module for efficient token allocation while preserving key pedestrian details. Second, missing spatiotemporal priors hinder consistent cross frame reasoning. ViCross mitigates this with a Spatial Constraint Enhancement Strategy that captures past motion, future locations, and action semantics for training-time proactive spatial rectification. Extensive experiments show that ViCross delivers clear gains in vision-centric prediction settings and is competitive with multi-source fusion approaches in several settings. Code is available at https://github.com/2tianyao1/ViCross.git.