Your Model Already Knows: Attention-Guided Safety Filter for Vision-Language-Action Models
2026-06-08 • Robotics
RoboticsMachine Learning
AI summaryⓘ
The authors found that certain parts of vision-language-action (VLA) models can naturally identify the object a robot should interact with, without extra training. They use this to keep the robot safe by treating everything else in the scene as obstacles and avoiding collisions, even if objects move around. Their method works in real-time and performs better than previous approaches that only check for obstacles once at the start. This shows that safety information is already inside these models and can be used directly for better robot control.
Vision-Language-Action modelsrobotic manipulationcollision avoidanceattention headsControl Barrier Functionreal-time object trackingvision-language modelsafety filterdynamic obstacles
Authors
Seongbin Park, Fan Zhang, Baharan Mirzasoleiman, Shahriar Talebi, Nader Sehatbakhsh
Abstract
Vision-Language-Action (VLA) models have demonstrated impressive end-to-end performance across a variety of robotic manipulation tasks. However, these policies offer no guarantees against collisions with task-irrelevant objects in the scene. Existing safety filters sidestep this problem by querying a vision-language model (VLM) to identify obstacles and their locations. This, however, is too slow to run in the control loop and can only be invoked at episode initialization, leaving the filter unable to track moving obstacles. We discover that a small number of attention heads within a VLA model reliably localize the object the policy intends to approach. These heads can be exploited within a training-free safety framework that obtains the active target from the attention heads at every step, treats the remainder of the scene as obstacles, and feeds these into a Control Barrier Function (CBF) filter. Together with a lightweight real-time object tracker, this allows for collision avoidance for non-static obstacles. We evaluate our framework on SafeLIBERO, which we extend with moving obstacles. On the original static benchmark, our method performs comparably to an oracle that uses privileged simulator state to identify the target, emulating a VLM-based identification step run once at episode initialization. On the dynamic variant, where the oracle's init-time target assignment becomes stale, our method substantially outperforms it by 43%, on average. Our findings suggest that the perceptual signals needed for real-time safety filtering are already present within VLA policies and can be exploited without additional training or heavy auxiliary models.