Autonomous driving system learns interactive traffic behavior from videos
INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving
Computer Vision and Pattern Recognition
Summary
Safe self-driving cars need to not only see what's around them but also predict how other cars and pedestrians will move next. The authors propose a method called INTERACT that learns to understand and predict the behavior of different objects in traffic from regular driving videos without needing labeled data. This method groups and tracks objects over time, figures out how the car’s own actions influence the scene, and uses this knowledge to plan safer driving paths. Their experiments show this approach improves safety by better anticipating interactions with other traffic participants.
What this means in practice
- •For autonomous vehicle engineers: Enhance trajectory planning by integrating interaction-aware predictions learned from video data without object labels, improving response to dynamic traffic scenarios.
- •For traffic simulation developers: Create realistic simulations of traffic dynamics by incorporating learned interaction-aware world models derived from real driving videos.
Authors
Zhan Chen, Songyan Zhang, Wenhui Huang, Zirui Li, Zhidong Wang, Qihang Huang, Shiyu Fang, Chen Lv
Abstract
Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.