World action models improve robot robustness against visual changes
PICA-WAM: Visual Perturbation Invariance via Cross-Appearance Self-Alignment in World Action Models
Computer Vision and Pattern RecognitionRobotics
Summary
World Action Models help robots predict future video and generate actions but often get confused by irrelevant visual changes, like different lighting or colors. The authors found that normal training doesn’t teach these models to ignore such changes when deciding actions. To fix this, they introduced a new training method that makes the model learn to act the same even if the robot sees visual differences. This method improved robot performance and made its behavior more stable in tests with simulations and a real robot.
What this means in practice
- •For robotics engineers: Make robots act reliably despite irrelevant visual changes like lighting or background variations by improving their world action models.
- •For autonomous vehicle developers: Enhance vehicle perception systems to be robust against visual disturbances without losing decision-making accuracy, using self-alignment training techniques.
Authors
Yude Zou, Yifei Yao, Guanbo Jiao, Jianming Ma, Guanjie Zheng, Xing Gao
Abstract
World Action Models (WAMs) jointly predict future video and generate actions, yet remain sensitive to task-irrelevant visual perturbations, i.e., changes in visual appearance that should not affect robot behavior. We identify a gap in conventional WAM training: video prediction encourages representations to preserve appearance information, while action generation does not explicitly enforce invariance to task-irrelevant appearance. Data augmentation exposes WAMs to such perturbations but does not require perturbed and nominal observations to produce consistent action representations, and alignment with external pretrained features does not guarantee the desired invariance. To address this gap, we propose PICA-WAM, a self-supervised training framework that casts perturbation invariance as an auxiliary cross-appearance self-alignment task for WAMs. An exponential-moving-average (EMA) teacher processes the nominal observation to provide a deeper-layer target, while the online WAM processes a perturbed observation and aligns its shallower-layer action representation to this target through a lightweight projection head. Across four strong WAMs evaluated on RoboTwin 2.0, LIBERO-Plus, EBench, and a real robot, PICA-WAM consistently improves robustness under visual perturbations, preserves nominal performance in simulation, and raises the real robot success rate. PICA-WAM further reduces the sensitivity of video features to task-irrelevant appearance and its propagation into actions, as shown by paired layer-wise diagnostics. Code: https://github.com/YudeZou/PICA-WAM-release