APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
2026-06-10 • Robotics
Robotics
AI summaryⓘ
The authors studied models that guide robots using both vision and language to perform actions. They found that these models struggle to understand new or unusual language instructions well because the training data has less variety in language compared to visuals and actions. To fix this, they split the learning process into two steps: first, they train the robot to act based only on vision and actions without language; then, they add language understanding carefully. Their method, APT, improves how well the robots follow new language instructions without messing up what they learned about actions.
Vision-Language Models (VLMs)Continuous action expertsOut-of-distribution (OOD)Language-agnostic priorGated fusion mechanismVisuomotor policyAction expert pretrainingCompositional tasksVision-Action (VA) prior
Authors
Kechun Xu, Zhenjie Zhu, Anzhe Chen, Rong Xiong, Yue Wang
Abstract
Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the $π$ and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/