Humanoid robot learns to act in real homes from simulation data
Perceptive Humanoid Motion from Simulation
RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Humanoid robots can do many physical tasks, but teaching them to move and understand instructions is hard because it requires lots of paired data linking what they see and hear to how they should move. The authors created HALO, a system that learns by simulating thousands of homes instead of relying on humans to provide data. HALO mastered many physical actions and combined them to complete longer tasks. Impressively, HALO worked well in real homes without having seen any real-world data during training.
What this means in practice
- •For robotics engineers: Build robots that learn diverse motions and language understanding purely from simulated home environments for real-world deployment.
- •For home automation developers: Develop systems that allow humanoid robots to perform household tasks autonomously in varied and unmapped real homes.$Commercial implications: Enables commercial deployment of autonomous household robots that adapt to real homes without manual data collection.
Authors
Siyuan Hu, Abulikemu Abuduweili, Yue Meng, Shipeng Liu, Quentin Leboutet, Chen Chen, Matthias Müller, Vladlen Koltun
Abstract
Humanoid robots have the potential to automate a wide range of physical tasks. Progress toward generalist humanoid autonomy, however, remains constrained by a data bottleneck: mapping pixels and free-form language to low-level joint commands requires paired supervision, and no corpus of such data exists at sufficiently large scale. Some approaches rely on human teleoperation and wearable capture devices, which require at least an hour of human time to produce an hour of data. We present HALO, a humanoid autonomy system powered by a pipeline that synthesizes vision-language-motion data across a thousand procedurally generated homes. The corpus scales with compute rather than human labor. HALO acquires a repertoire of physical skills, including sitting on chairs, navigating to named objects, stepping sideways through half-open doors, crouching and crawling beneath tables, and pushing boxes. It orchestrates these skills to solve composite, long-horizon tasks. Without ever seeing a real home during training, HALO transfers zero-shot to the physical world, achieving a 91% success rate across 360 trials in six unmapped real-world environments. See https://apple-aiml-research.github.io/ml-halo/ for code and videos.