FailPatch improves vision language action models by fixing failure states
FailPatch: Failure Residual Patching for Vision-Language-Action Models
Robotics
Summary
Robots and AI systems that follow vision and language instructions often learn from examples where they succeed, but rarely get training on how to handle situations where they fail. The authors propose FailPatch, a method that separately learns when a robot's actions are reliable and when they might fail, allowing it to adjust behavior specifically in those failure states. They add small specialized modules to an existing system to help it recognize and correct failures without changing the whole model. This approach improved performance on several simulated and real-world robot tasks by helping the model recover from mistakes better.
What this means in practice
- •For robotics engineers: Improve robot control systems by directly addressing failure-prone situations to boost task success rates in complex environments.
- •For industrial automation teams: Enhance vision-language guided automation processes to better handle unpredictable conditions by patching failures without full retraining.
- •For self-driving car developers: Use failure-driven residual experts to improve policy reliability when encountering rare or unexpected driving scenarios.$Commercial implications: Enables safer and more robust autonomous vehicle control by specifically correcting failure states, critical for commercial deployment.
Authors
Peng Yu, Jiacheng Wang, Ziheng Zhang, Xuchong Zhang, Baoting Li, Zhuoyuan Yu, Yuxiang Chen, Tiancai Wang, Hongbin Sun
Abstract
Vision-Language-Action (VLA) policies are typically adapted using successful demonstrations, which provide direct action supervision but rarely cover failure-prone states. Deployment failures expose these states, yet lack the corrective actions needed for conventional supervised learning. We propose FailPatch, a failure-driven residual patching framework that decouples action supervision from execution-reliability supervision. Successful demonstrations ground how the policy should act, while deployment trajectories indicate when its behavior becomes unreliable. We further observe that action hidden representations exhibit clear linear separability between reliable and failure-associated states while directly conditioning action generation. Building on these insights, FailPatch introduces a Null-gated Residual Expert Bank into the action hidden space of a frozen VLA policy. A unified Preserve--Redirect--Trust objective retains the original policy in reliable states, selects residual experts in failure-associated states and redirects representations from failure regions toward success-associated regions under bounded intervention. With only 0.52% trainable parameters, FailPatch improves success rates by 11.0 percentage points on four long-horizon RoboTwin tasks under clean evaluation, 9.5 percentage points under clean-to-random generalization, and 16.7 percentage points over the baseline across three real-world tasks. Project and code: https://github.com/yupeng-2003/FailPatch.