Driving models plan continuous paths inside visual language networks
Planning in the Backbone: DiffAdapterVLA for Native Continuous Trajectory Generation with Driving VLMs
RoboticsArtificial Intelligence
Summary
Generating smooth driving paths usually happens separately from how driving conditions are understood by vision-language models. The authors introduce DiffAdapterVLA, which mixes trajectory planning directly into the driving model’s internal layers. This lets the model refine planned routes step-by-step, adapting continuously as it processes visual and context information. Their approach achieves better, faster trajectory planning using fewer new parts added to existing models.
What this means in practice
- •For autonomous vehicle engineers: Integrate continuous trajectory planning inside driving perception models to enable faster and more efficient route generation during vehicle operation.
- •For robotics software developers: Incorporate planning modules into vision-language backbones to improve real-time path generation in robots navigating complex environments.
Authors
Changxin Lu, Xiaoliang Meng, Yu Wu, Rui Huang, Honglin Li, Tao Chen, Kaixuan Zhou, Yadong Shao
Abstract
Pretrained driving vision-language models (VLMs) integrate visual, route, language, and driving context into rich driving priors, yet their representation objectives remain separated from continuous driving planning. Existing methods typically begin trajectory generation only after the VLM has formed a final condition, leaving depth-wise condition computation outside the stepwise formation of trajectory state. We introduce DiffAdapterVLA, which realizes Planning in the Backbone: it injects explicit trajectory tokens into selected VLM late layers, bringing trajectory state into backbone forward computation, where it co-evolves with driving conditions at different depths. Lightweight layer-wise DiffAdapters organize this computation into recursive trajectory refinement, while asymmetric joint attention preserves directed guidance from the condition stream to trajectory planning. By placing planning within existing backbone computation rather than relying on an independent trajectory planner, DiffAdapterVLA adapts only lightweight trajectory modules to turn existing driving priors into efficient continuous planning capability. NAVSIM results show that it achieves high-quality closed-loop planning with low end-to-end latency using few trainable parameters, and demonstrate that jointly evolving trajectory state and depth-wise driving conditions in VLM late-layer computation effectively realizes continuous trajectory planning.