Hierarchical vision language action model improves robot manipulation tasks
H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space
Robotics
Summary
Robots often struggle to follow complex instructions because they directly connect what they see and hear to detailed movements, which is tricky when things change. The authors propose a new system called H-VLA that first figures out key action steps and then plans detailed movements, separating thinking from doing. This makes the robot better at understanding instructions and adapting to different scenes or camera views. Tests show H-VLA performs well both in simulations and with real robots, especially when handling new or changing environments.
What this means in practice
- •For industrial robotics teams: Improve robot control in factories by separating high-level action planning from low-level motion, enabling better adaptation to new tasks and environments.
- •For robotics software developers: Develop software that integrates language understanding with motion generation using a unified action representation across different robots and viewpoints.
Authors
Xiongfeng Peng, Lu Xu, Yandong Wang, Jiaqian Yu, Zirui Zheng, Yamin Mao, Weiming Li, Inseop Chung, Hyun-woong Cho, Jaewook Yoo, Dongwook Lee, Daehyun Ji, Chao Zhang
Abstract
Vision-Language-Action (VLA) models have shown strong potential for robotic manipulation, but many existing methods still rely on direct mappings from language and visual observations to dense actions. This formulation can weaken the semantic reasoning capability inherited from pre-trained Vision-Language Models (VLMs), which are mainly optimized for visual-linguistic understanding rather than low-level control, and becomes fragile under spatial variations, including changes in object positions, scene layouts, robot embodiments, and camera viewpoints. To address these limitations, we propose H-VLA, a hierarchical VLA framework that decouples high-level key-action reasoning from low-level motion generation. H-VLA combines a Key-Action Model for predicting a key-action as the next manipulation subgoal, a Motion Planning Model for generating dense future actions conditioned on the predicted key-action, and a Unified Camera-Centric Action Space for consistent representation across datasets, embodiments, and viewpoints. We further adopt a two-stage training strategy that emphasizes key-action reasoning during pre-training and dense motion generation during fine-tuning. Experiments show that H-VLA achieves strong performance on SimplerEnv, reaching 91% on Google Robot visual matching, 84% on Google Robot variant aggregation, and 81% on WidowX visual matching. On Agilex real-robot tasks, H-VLA improves over the strongest baseline by 10, 47, and 16 percentage points under in-distribution, out-of-distribution position, and out-of-distribution scene/object settings, respectively.