Vision language action models get better at spotting and classifying failures early
Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing
RoboticsArtificial Intelligence
Summary
Vision-Language-Action models help robots and AI systems do tasks using images and language. But when these models encounter new situations, they can fail in dangerous or costly ways. The authors created a tool called VLA-Tug that watches these models from inside and can spot problems early with few mistakes. It also tells what type of failure is happening, so the system can try to fix it. This method works faster and more reliably than previous tools, even on new tasks.
What this means in practice
- •For robotics engineers: Use VLA-Tug to monitor robot vision-language systems and detect failures early during manipulation tasks to prevent damage or accidents.
- •For industrial automation teams: Deploy lightweight failure monitors on edge devices managing vision-language control to improve safety and task success under unexpected conditions.
Authors
Michal P. Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P. Bhatt, Matthias Scheutz, Ufuk Topcu
Abstract
Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.