Transformer model improves pedestrian crossing predictions for autonomous vehicles
TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs
Computer Vision and Pattern Recognition
Summary
Predicting if a pedestrian will cross the road is important for self-driving cars to stay safe. The authors developed TrajFusionNet+, a new computer model that uses three kinds of information: past pedestrian movements, visuals of those movements overlaid on street scenes, and relationships between pedestrians and nearby traffic elements. This model combines all these views using transformers, a type of machine learning technique. It outperforms previous methods on popular pedestrian datasets and is better at handling different datasets, showing it can generalize well to new situations.
What this means in practice
- •For autonomous vehicle developers: Enhance pedestrian crossing predictions by integrating trajectory, visual, and relational scene data for safer driving decisions.$Commercial implications: Enables safer autonomous vehicle systems with improved pedestrian behavior anticipation, creating valuable competitive advantage.
- •For smart city traffic management teams: Improve pedestrian flow prediction by fusing multiple data sources about pedestrian movement and interactions with traffic elements in real time.
Authors
François G. Landry, Moulay A. Akhloufi
Abstract
The pedestrian crossing intention task involves predicting whether pedestrians are likely to cross the road from the point of view of an autonomous vehicle. We introduce TrajFusionNet+, a novel transformer-based model for pedestrian crossing intention prediction. TrajFusionNet+ combines sequential and visual representations of pedestrian trajectory with a graph-based representation of the scene context in order to predict pedestrian crossing intention. The proposed architecture builds upon our previous model, TrajFusionNet, and comprises three branches: a Sequence Attention Module (SAM), which processes a sequential representation of past and predicted pedestrian trajectories; a Visual Attention Module (VAM), which utilizes a visual representation of the pedestrian trajectories by overlaying observed and predicted bounding boxes onto scene images; and a Graph Attention Module (GAM), which extracts pedestrian-centric graphs from segmented scene images and captures the relational dependencies between pedestrians and traffic elements. TrajFusionNet+ achieves improved state-of-the-art performance on the two most widely used pedestrian crossing intention datasets, PIE and JAAD. Furthermore, we introduce a new evaluation protocol in which models are trained jointly on the PIE and JAAD datasets but evaluated separately on each. Under this setting, TrajFusionNet+ demonstrates superior generalization compared to existing approaches.