Vision language models cut driving effort by half with smart scene focus
Scene-Adaptive VLA: Efficient Autonomous Driving via Dynamic Layer Routing
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
Driving autonomously with AI that understands both images and language often uses a lot of computer power, which drains energy quickly. The authors created a method that adjusts how much of the AI's brain it uses depending on what's happening in the driving scene. This way, the AI works less when the scene is simple and uses more computing power only when needed, saving energy without making more mistakes. They tested this on a driving benchmark and showed it cuts computation by over half with almost no loss in driving ability or explanation quality.
What this means in practice
- •For autonomous vehicle engineers: Reduce the real-time computation and power consumption of AI driving systems by dynamically adjusting model complexity based on scene complexity.
- •For edge device developers: Design efficient AI systems for resource-limited devices by implementing scene-adaptive dynamic model execution to balance performance and energy use.
Authors
Yujia Yang, Quan Yuan, Jiawei Fan, Yang Li, Tiange Fu, Xiaoyuan Fu, Guiyang Luo, Jinglin Li
Abstract
Vision-Language-Action (VLA) models offer a promising paradigm for autonomous driving. However, the massive parameter scale of VLAs strains onboard computational resources during real-time inference. This directly increases energy consumption and limits vehicle battery range, making it critical to minimize computational overhead without compromising driving capability and explainability. Current research on efficient VLA-based driving focuses on sequential adaptation (\textit{e.g.}, Chain-of-Thought and token adjustment), leaving structural adaptation underexplored despite parameter redundancy across dynamic driving scenes. To address this gap, we propose Scene-Adaptive VLA, an efficient framework leveraging dynamic layer routing to adjust active parameters based on real-time scenes. Inspired by the temporal continuity of scenes and model predictive uncertainty, our approach characterizes model-perceived scene complexity via frame-level temporal scene variations and decision deviations. To uncover latent scene-to-parameter correspondences, we encode this characterization into scene-aware tokens, alongside two specialized queries. A budget-conditioned layer routing mechanism evaluates these refined queries to allocate a budget that determines the active parameter ratio and subsequently selects appropriate layer combinations. The entire framework is optimized end-to-end. Extensive evaluations on the Bench2Drive benchmark show that when adapted to a state-of-the-art VLA model, our approach significantly reduces inference-time computational overhead by 53.5\% with negligible degradation in driving and language capabilities. Code is available at https:github.com/yang-luvu/Scene-Adaptive-VLA.