Vision language driving models sense hazards but fail to brake safely
Perceives but Doesn't Brake: A Perception-Action Binding Gap in Reasoning Driving VLAs
RoboticsArtificial Intelligence
Summary
Some self-driving car systems can recognize dangers on the road but don’t always slow down when they should. The authors found that while the model knows about hazards, this information doesn’t always affect its driving decisions, especially the braking action. They showed that this problem comes from a weak link between seeing the hazard and acting on it. By adding a simple adjustment during the decision-making step, they helped the model brake properly when needed and avoid crashes.
What this means in practice
- •For autonomous vehicle developers: Improve driving safety by fixing how models use hazard info to generate braking trajectories in simulated test environments.
- •For robotics system integrators: Enhance robot navigation safety by integrating hazard-conditioned control signals into trajectory planning without retraining perception models.
Authors
Matteo Cercola, Katie Z Luo, Simone Formentin, Marco Pavone
Abstract
Reasoning-augmented vision-language-action (VLA) driving models can perceive and reason about safety-critical hazards yet fail to translate that evidence into safe actions. We characterize this failure as a perception-action binding gap: hazard information is available inside the model but exerts insufficient control over trajectory generation. Using matched simulated counterfactuals and logged driving replay, we show that in-path hazards are actually encoded in the visual representation, but longitudinal plans are dominated by ego-speed history and respond only weakly to hazard proximity or the model's own reasoning trace. Further analysis shows that the trajectory generator is capable of braking but sometimes fails to do so due to the perception-action binding gap. Without retraining or modifying any weights, we inject a hazard-conditioned energy into the frozen flow-matching sampler of Alpamayo-1.5, restoring the missing coupling at generation time. With exact obstacle geometry, guidance cuts footprint intrusion from 58.1% to 14.5% and recovers safe braking whenever sufficient stopping distance remains, eliminating collisions entirely in our braking-relevant closed-loop battery. It also reverses the diagnosed dependence of the plan, from tracking ego speed to responding to obstacle range. These results separate hazard perception from hazard control in driving VLAs, show that unsafe behavior can arise even when the model has both the evidence and the ability to act on it, and propose a simple fix for bridging this binding gap.