Vision language action models monitor themselves to avoid failures
Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models
Robotics
Summary
Vision-language-action models help robots perform tasks by understanding pictures and instructions, but they can make mistakes. The paper shows that instead of using extra big models to check if the robot is doing well, you can watch what's happening inside the robot’s own brain. By training simple monitors on the robot's internal signals during normal tasks, the system can tell if something is going wrong. This new way works as well or better than older methods and is less complicated and costly.
What this means in practice
- •For robotics engineers: Deploy robot control systems that internally detect and flag execution mistakes without needing extra supervision models.
- •For industrial automation teams: Implement cost-effective runtime monitoring in automated factories by using existing robot model signals instead of additional monitoring software.
Authors
Atiksh Bhardwaj, Edward W. Duan, Junwon Seo, Prithwish Dan, Wei-Chiu Ma, Andrea Bajscy, Preston Culbertson
Abstract
Vision-Language-Action (VLA) models can perform diverse manipulation tasks directly from visual observations and natural language instructions, but they remain prone to mistakes and unpredictable failures. Deploying these policies without continuous human supervision will require robust runtime monitors that detect when the policy has gone off-course. Existing monitors use external foundation models, such as vision-language models (VLMs) to supervise the policy, introducing significant complexity and computational cost for monitoring at runtime. In this work, we investigate an alternate approach: \textit{can a VLA's own internal activations provide the information needed to monitor its execution?} Our method trains linear probes on policy activations from demonstrations to estimate progress, then flags executions that deviate from expected progress. In experiments with the open-source VLA $π_{0.5}$ and across simulated and real-world manipulation tasks, this approach is competitive with or outperforms VLM-based monitors in task failure detection, as well as methods which require failure data for training. These findings suggest that simple probes of VLA internals can be used to develop reliable runtime monitors, opening a path toward building policies that can act reliably and monitor their execution without external foundation models.