Summary
Controlling humanoid robots to move and use their bodies effectively is hard because a single command often must handle both what the robot wants to do and how it moves its whole body. The authors show a way to separate these by focusing on a few important body poses and combining those with learned motion patterns to represent both what the robot should do and how to move smoothly. Their system uses a model that predicts key body poses from vision and language instructions and then generates full-body motion in real time, improving the robot’s ability to perform complex tasks involving moving around and using its hands. They tested this system both in simulations and on a real robot, and it worked better than older methods.
What this means in practice
- •For robotics engineers: Control humanoid robots more effectively by using sparse critical poses combined with learned motion priors for real-time loco-manipulation tasks.
- •For automation integrators: Implement improved whole-body coordination in service or research humanoid robots that interact with diverse environments and objects.
Authors
Zhaohui Yang, Quanbao Lin, Chengqi Shi, Zhenglang Weng, Jicheng Lin, Boyu Chen, Tengjie Zhu, Guanzhu Ren, Jiahao Xu, Yang Feng, Xiaokang Yang, Yao Mu
Abstract
Humanoid loco-manipulation requires an effective interface between high-level task policies and whole-body control. Existing interfaces typically rely on a single action representation for both task intent and whole-body coordination. This may burden the high-level policy or leave one aspect underspecified. We present SP-HLM, a framework built on a sparse-pose-driven interface. The interface combines five task-critical body poses with a learned latent motion prior to represent spatial intent and whole-body coordination in complementary forms. SP-VLA, a vision-language-action model, first predicts task-relevant sparse poses and finger joint positions from multimodal observations and language instructions using asynchronous action-chunk inference. Given these sparse targets, SP-MGM, a motion generation model trained through two-stage distillation, reconstructs coordinated full-body motion in real time. A pretrained motion encoder maps this motion to a compact latent representation. SP-GMT, a general motion tracker, then integrates the explicit sparse targets with this implicit motion prior for whole-body control. Experiments in simulation and on a real humanoid robot demonstrate improved motion tracking, whole-body coordination, and loco-manipulation success over existing approaches. Additional results and videos are available on our \href{https://sp-hlm.github.io/}{\textcolor{blue}{project page}}.