Humanoid robot learns whole-body skills to close doors autonomously
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
Robotics
Summary
Humanoid robots find it hard to learn how to move and use their whole body together, especially when touching and interacting with objects like doors. The authors created ViLoMan, a system that turns partial human demonstrations into full robot movements and teaches the robot to use only its own sensors to perform tasks. ViLoMan lets a robot close different types of doors from various starting positions without help from outside information. This approach was tested successfully both in computer simulations and in the real world.
What this means in practice
- •For robotics engineers: Create humanoid robots that autonomously perform complex tasks like closing doors using only onboard sensors and whole-body motion control.
- •For robotics simulation developers: Develop simulation environments that generate complete humanoid robot trajectories from partial human demonstrations to train control policies.
Authors
Zejie Tian, Ruibing Hou, Bingpeng Ma, Börje F. Karlsson, Shiguang Shan
Abstract
Humanoid loco-manipulation requires adaptive whole-body coordination to seamlessly integrate locomotion and physical interaction. Despite recent advances, learning autonomous loco-manipulation remains challenging due to the scarcity of diverse, physically executable robot-object interaction data and the difficulty of learning unified whole-body control directly from onboard observations. We present ViLoMan, a scalable framework for autonomous humanoid loco-manipulation. ViLoMan first transforms partial kinematic demonstrations of human-object interactions into complete, physically executable robot trajectories. It then leverages these trajectories within a teacher-student distillation framework to learn a unified policy that maps egocentric depth observations and proprioceptive measurements directly to joint-level whole-body actions. During deployment, the policy requires neither reference motions nor intermediate commands. We evaluate ViLoMan on door-closing tasks across diverse door configurations and robot initial conditions in both simulation and the real world. Experimental results demonstrate that a single policy enables a Unitree G1 humanoid to complete the full task using only onboard depth sensing and proprioception, while generalizing robustly across task variations and transferring effectively from simulation to reality. Project page: viloman-anonymous.pages.dev.