Humanoid robots improve walking by learning to handle missing information
RoboDreamer: Anticipatory Humanoid Locomotion with Predictive State-Space Models
Robotics
Summary
Humanoid robots need to keep walking steadily even if their sensors don’t always give complete or perfect information. The authors developed RoboDreamer, which trains a robot first with clear data, then teaches it to guess missing details by looking at past information. This helps the robot stay balanced and move smoothly, even when some information is hidden or incomplete. Their approach worked well in computer simulations and in real robots, showing more reliable walking and quick responses.
humanoid locomotioncontrol policysensor noisetemporal contextteacher-student trainingstate-space modelaction refinementIsaacLabMuJoCoUnitree G1
Authors
Zhe Li, Yangyang Wei, Xichen Yuan, Zhenzhe Zhang, Weihao Yuan, Shanghang Zhang, Jianfei Yang
Abstract
Humanoid locomotion requires control policies that remain stable under imperfect sensing while exploiting temporal context for consistent motion. We present RoboDreamer, a two-stage teacher--student framework that combines next-observation consistency with randomized continuous temporal masking. A teacher is first trained on clean observations, and a student is then distilled under masked recent observations, encouraging the policy to infer missing current information from history. At inference, the same masking interface is reused for implicit closed-loop action refinement and optional multi-step action chunking. Mamba is used as the temporal backbone, while matched ablations show that masking/distillation provides a substantial part of the gain and Mamba contributes additional tracking improvements with real-time latency. Experiments in IsaacLab, MuJoCo, and on a Unitree G1 demonstrate robust motion tracking under observation masking and successful real-world deployment.