VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
2026-07-06 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors propose VLM-CASE, a new system to help self-driving cars drive more safely in bad weather by anticipating road conditions before problems happen. They use a vision-language model (VLM) that looks at camera images to understand the road's condition and adjusts how the car drives based on that, keeping it within safe limits. Their system plans the car's movements to stay safe while adapting to changing grip and visibility, unlike other systems that only react after seeing a problem. They tested this approach in a driving simulator and found it worked better than current methods. This framework can work with many types of low-level controllers, making it flexible for future autonomous driving systems.
autonomous drivingvision-language modellow-rank adaptationcontext-adaptive safety envelopefriction budgetmodel predictive controlresponsibility-sensitive safetyanticipatory controlclosed-loop simulationCARLA simulator
Authors
Tianjia Yang, Ke Li, Ruwen Qin, Xianbiao Hu
Abstract
Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation, reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A vision-language model (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.