ForeDrive improves autonomous driving planning using future-focused visual models
ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model
Computer Vision and Pattern Recognition
Summary
Self-driving cars need to predict the world around them to plan safe paths, but not all prediction methods help with planning. The authors developed ForeDrive, a system that learns to represent future road conditions in a way that directly helps plan driving routes. It uses a specialized model that focuses on useful future information and combines it carefully with current visual input to make decisions. ForeDrive performs well using only front camera images and imitation learning, without complex extra training steps.
What this means in practice
- •For autonomous vehicle engineers: Use ForeDrive's planning-relevant latent representation to improve path planning from front camera images without complex extra training.
- •For robotics software developers: Incorporate ForeDrive's method of combining predicted futures with current views for trajectory generation in robots navigating dynamic environments.
Authors
Sinuo Wang, Zichong Gu, Yuhan Huang, Wenxin Wen, Xun Yang, Yiqing Zhang, Xingyu Zhang, Ningyu Che, Jie Ling, Qiankun Yu, Wei Liu, Jing Xu, Xinggang Wang
Abstract
Existing latent world models are typically optimized for future predictability, yet the resulting representations are not necessarily useful for planning in autonomous driving. Predictions are commonly used for pretraining or auxiliary supervision rather than as direct conditioning signals for trajectory generation. We propose ForeDrive, which learns a planning-relevant latent representation and couples it asymmetrically to a Diffusion Transformer (DiT) planner. The planner consumes multi-horizon latent future representations learned with a JEPA-style world model; planning gradients update the shared online encoder, while stop-gradient routing trains the latent predictor with forecasting losses only. Because predicted futures have varying reliability across horizons and BEV trajectories are misaligned with image tokens, we use gated visual fusion, future-status injection, and Trajectory-Adaptive Bias (TAB) to inject future latents as guidance without overriding the current observation. Trained with pure imitation learning and using only the current front-view image as visual input at inference, ForeDrive attains 89.9 PDMS on NAVSIM v1 and 90.0 one-stage EPDMS on NAVSIM v2, without reinforcement learning or an external trajectory scorer.