General Action Expert enables real-time humanoid robot teleoperation
GAE: General Action Expert for Real-Time Humanoid Teleoperation
Robotics
Summary
Controlling humanoid robots in real time to imitate human movements is difficult because of delays and differences between human and robot bodies. The authors created a method called General Action Expert (GAE) that trains robots to copy a wide variety of human motions smoothly and quickly. They collected motion data from videos, animations, and motion capture, then taught their system to predict and execute robot motions even when there is a communication delay. Tests on real robots show that GAE can make robots move naturally and respond fast to a human operator's actions.
What this means in practice
- •For robotics engineers: Create teleoperation systems that enable humanoid robots to mimic diverse human movements with low delay and high fidelity in real time.
- •For remote service operators: Operate humanoid robots remotely to perform complex social or labor tasks by closely synchronizing robot actions with human gestures.
Authors
Yuefan Wang, Huaicheng Zhou, Xiao He, Zhijie He, Mingchuan Yang, Huayi Zhang, Li Chai, Jinxin Liu, Donglin Wang
Abstract
Humanoid avatars extend human physical presence beyond the body, enabling people to participate in social, service, and labor activities through remotely operated robots. This requires teleoperation systems capable of realizing diverse and dynamic whole-body behaviors while maintaining responsive human-robot synchronization. We present General Action Expert(GAE), a unified learning framework for general-purpose, low-latency humanoid whole-body teleoperation. To cover diverse human behaviors, GAE builds a large-scale human motion dataset from heterogeneous sources, including videos, animations, and motion capture, followed by standardization and augmentation. GAE then addresses the noise and embodiment mismatch in human motions with a two-stage training paradigm: a privileged generator policy first tracks human motion references in simulation and rolls out feasible humanoid trajectories; a deployable executor policy then learns to track these generated trajectories under curriculum domain randomization. For responsive human-robot synchronization, GAE introduces a latency-conditioned anticipation mechanism that adaptively compensates for end-to-end delay during real-time teleoperation. Simulation and real-world experiments on Unitree G1 and Westlake O1 robots demonstrate that GAE enables humanoids to smoothly mirror diverse, agile, and expressive human behaviors. Project website: https://wangyf0928.github.io/gae-wlrobotics/