Vision language action policies improve with modular advantage post training
Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies
Robotics
Summary
Robots that understand images, language, and actions need better ways to learn from limited experience. The authors studied a method called advantage-guided reinforcement learning to improve robot control using small amounts of data. They carefully tested different parts of this method separately to see what works best. Their improved recipe helped robots perform better on real tasks and showed ways to predict success without always testing on actual robots.
What this means in practice
- •For robotics software developers: Improve robot control policies by refining reinforcement learning with modular advantage post-training components informed by offline evaluation.
- •For industrial automation engineers: Enhance multi-arm robot manipulation in manufacturing using the identified training recipe for better task success with limited real-world data.
Authors
Jiahang Cao, Hanye Zhao, Hang Lai, Shenyu Zhang, Xiaoshen Han, Xinghang Li, Futeng Liu, Wanli Peng, Heyun Wang, Yunhong Wang, Jason Li, Yong Yu, Weinan Zhang
Abstract
Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.