Humanoid robot adapts whole-body motions to different terrains in teleoperation
NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation
Robotics
Summary
Controlling a robot to mimic a person’s movements is tricky when the robot walks on a different kind of ground than the person. The authors developed NEXUS, a system that lets a robot sense the terrain it is on and adjust its whole-body movements accordingly, instead of copying motions exactly. They created a large dataset of paired motions across terrains and trained the robot to follow human commands while adapting to new terrains. This approach works well even in new environments the robot has never seen before.
What this means in practice
- •For robotics engineers: Enable teleoperated humanoid robots to walk and balance on uneven or unknown terrain by combining human commands with robot sensory input.
- •For remote inspection teams: Operate robots remotely in disaster or construction sites with varied ground surfaces by adapting robot movement without needing terrain-specific tuning.
Authors
Xiangyu Miao, Junsong Wu, Jiyuan Shi, Weiji Xie, Jinrui Han, Xingyi Wang, Weinan Zhang, Chenjia Bai, Xuelong Li
Abstract
Whole-body teleoperation requires a humanoid robot to reproduce a human operator's behavior even when their terrains differ. This demands that the robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control. To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher-student learning to reproduce commanded behaviors across terrains. Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method. Project website: https://nexus-humanoid.github.io/