Vision language action models avoid visual shortcuts with task scrubbing
When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Robot learning can be tricked when models focus on obvious but useless visual clues, like backgrounds or camera angles, instead of the real task. The authors studied how different vision-language-action models handle this problem and found a way to measure their tendency to use these shortcuts. They introduced a training method called task scrubbing that helps models ignore misleading visual cues and rely more on language instructions. This method makes robot actions more reliable, especially when faced with new or different environments.
What this means in practice
- •For robotics engineers: Improve robot control policies to generalize better by reducing reliance on misleading visual features during training.
- •For autonomous system developers: Enhance vision-language-action models to perform reliably in new environments by applying task scrubbing to prevent shortcut learning.
Authors
Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu, Mohammad Mohammadi, Shuhong Zheng, Igor Gilitschenski
Abstract
Shortcut learning is a prevalent issue in robot learning. The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background. Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives. We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning. We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts. Visual shortcuts consistently enter action representations in early layers, with models differing in the extent to which later layers correct them by incorporating language information. To boost models' attention to language, we introduce task scrubbing, a new domain-adversarial training method that decreases models' likelihood of using visual shortcuts and improves VLAs' generalization. Experiments in both simulation and the real world across multiple VLAs and visual cues show that task scrubbing improves out-of-distribution robustness and often eliminates visual shortcut learning.