Vision language action models improve skill mix success with new training method
Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs
RoboticsMachine Learning
Summary
Vision-language-action models help robots or AI systems understand visual scenes and follow instructions to do tasks. However, they often fail when asked to perform new combinations of skills that they have not been specifically shown before. The authors found that these models can rely too much on visual clues instead of instructions during training, leading to errors. They developed a method called CRAFT that teaches the model to reuse known skills correctly even in unseen combinations by training with special example pairs. This approach improved the AI’s success when tested in simulations and on a real robot.
What this means in practice
- •For robotics developers: Enable robots to perform new combinations of tasks they were not directly trained on by better skill reuse during instruction following.
- •For autonomous system engineers: Improve control policies in autonomous agents that must generalize to unseen task mixes using vision and language input.
Authors
Taegeun Yang, Youngju Na, Yoonki Cho, Sung-Eui Yoon
Abstract
Vision-language-action (VLA) models often struggle to generalize to skill combinations absent from their fine-tuning demonstrations, even when every constituent skill has been demonstrated. We focus on a vision shortcut as one failure mode: during fine-tuning, visual observations can serve as a proxy for the instruction, so a policy may execute a demonstrated combination associated with similar observations rather than the instructed combination. This motivates training with counterfactual pairs formed by holding a demonstration observation fixed while changing the instruction to specify an undemonstrated combination. These pairs, however, lack corresponding demonstrated action targets. Crucially, the currently required skill has already been demonstrated, but actions from those executions cannot serve as direct targets because the same skill can require different actions across observations. We propose CRAFT, which transfers supervision from demonstrated executions of the required skill to counterfactual pairs using skill representations that can be reused across executions of the same skill. Across three VLA models and two simulation benchmarks, CRAFT improves success on undemonstrated combinations while maintaining high success on demonstrated ones; it also improves compositional generalization on a real robot. Project website: https://taegeunyang.github.io/craft/