Visual models improve autonomous driving but choice and training matter
Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?
RoboticsComputer Vision and Pattern Recognition
Summary
Using advanced visual models can help self-driving cars make better decisions on the road. The authors show that these visual tools improve driving performance across different driving software, even without extra training. However, which visual model is used matters a lot, and adding extra perception training can make the system less sensitive to this choice. They built a new driving system that performs better than recent ones by carefully selecting the visual model and training method.
What this means in practice
- •For autonomous vehicle developers: Improve self-driving car software by selecting and aligning visual models to boost driving performance without changing planner hardware.
- •For robotics system integrators: Integrate visual foundation models with perception supervision to create more robust autonomous navigation in varied environments.
Authors
Zihao Zhang, Haochen Tian, Tianyu Li, Changhui Jing, Jingliang He, Naisheng Ye, Ziyuan Pu, Zhenjie Yang
Abstract
Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance. To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged. Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation. (2) The choice of VFM target matters for planning performance, and alignment to a different VFM can further benefit planners with pre-trained VFM encoders. (3) Auxiliary perception supervision reduces sensitivity to VFM target selection, narrowing the EPDMS spread across five targets from 2.7 to 0.5 points and potentially compensating for less effective VFM targets. Guided by these findings, we develop ViRA-Diffusion, a diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points. The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving. The results and demo are available at https://github.com/OpenDriveLab/ViRA.