Self-adaptive vision language action model improves robot performance under hardware shifts
Self-Adaptive VLA for Robust Robot Deployment
RoboticsComputer Vision and Pattern Recognition
Summary
Robots that use vision, language, and action to perform tasks often struggle when their hardware changes or wears down over time. The authors developed a way for these robot policies to learn and adjust themselves during use without needing constant recalibration. They collect examples of the robot working under different hardware issues, then teach the robot to recognize and adapt to these changes by encoding context and adjusting behavior. This approach helps robots keep working well even when parts like actuators or sensors start to drift.
What this means in practice
- •For industrial robot operators: Enable robots to maintain high precision in manufacturing despite hardware wear or calibration issues using self-adaptive policies.
- •For robotic system integrators: Improve robustness of robot deployments across different workstations by allowing iterative hardware shift adaptation without manual recalibration.
Authors
Hongxin Zhang, Chunru Lin, Tsun-Hsuan Wang, Zhenjia Xu, Chuang Gan
Abstract
While Vision-Language-Action (VLA) models demonstrate impressive capabilities in robotic manipulation, their memoryless nature renders them brittle to test-time environment shifts, particularly hardware shifts caused by wear or imperfect calibration. Enabling these models to self-adapt during deployment without requiring continuous on-site recalibration remains a critical bottleneck for real-world scalability. In this work, we introduce Self-Adaptive VLA, a novel post-training recipe that enables the policy to iteratively adapt to deployment-time hardware shifts leveraging its own rollouts as context. To do so, we first collect policy rollouts under deliberately injected hardware shifts. We then transform the base policy's training data into shift-conditioned expert demonstrations by pre-compensating the expert actions for these known shifts. Next, we introduce a lightweight, plug-in context encoder that compresses the context, including visual observation, proprioception, and actions in the shifted environment, into a latent context token. This token modulates the policy through adaptive layer normalization (AdaLN). Furthermore, we find that context tokens can be ensembled, allowing the policy to iteratively self-correct and mitigate failures step by step. Extensive experiments across four precision-critical bi-manual and dexterous manipulation tasks show that Self-Adaptive VLA recovers over 80% of the base policy's performance under hardware shifts, such as actuation bias and joint encoder offsets. Moreover, Self-Adaptive VLA enables more robust deployment to new workstations compared to the base policy. Our approach provides a pathway for robust large-scale real-world robot deployments and easier maintenance. See videos at https://icefoxzhx.github.io/self-adaptive-vla.