Humanoid robot successfully picks up and moves objects using onboard vision and sensing
ReGATE: Recurrent Gated Visual Fusion for Reactive Humanoid-Object Interaction
Robotics
Summary
Robots that have human-like bodies need to grab, carry, and put down objects while moving around and reacting to partial or missing visual information. The authors created ReGATE, a system that combines depth vision from the robot’s own perspective with its sense of body position to better predict object shapes and contacts. It uses a special learning method to improve object handling even when the robot can’t see well or at all during the action. Tests show ReGATE is much better at grabbing objects than previous methods and works on both simulations and a real robot.
What this means in practice
- •For robotics engineers: Enable humanoid robots to reliably grab, transport, and release objects even with incomplete or interrupted onboard vision.
- •For warehouse automation teams: Use a humanoid robot controller that adapts to moving and occluded objects for improved handling in cluttered settings.
Authors
Hao Zhu, Jorge Peña-Queralta
Abstract
Humanoid loco-manipulation requires coordinating object acquisition, transport, and release from onboard sensing while responding to user commands and incomplete visual observations. We present ReGATE, a whole-body controller that combines egocentric depth, proprioception, and online locomotion and contact commands. Built on a teacher-student framework, ReGATE integrates interaction-state estimation through a recurrent adapter that predicts object geometry and physical contact without an external object-pose estimator. Head-wise and global gates regulate visual updates to a proprioceptive pathway. A visual-grounding and corruption curriculum trains the adapter, while phase-aware initialization increases coverage of brief contact transitions. In simulation, ReGATE achieves 85.2% success across 330 static-object acquisition trials, compared with 48.5% for VisualMimic under the same protocol, and 85.9% across 440 moving-object acquisition trials. Deployment on a Unitree G1 demonstrates commanded grasp-transport-release sequences and acquisition of laterally displaced boxes. A qualitative hardware experiment further demonstrates successful box acquisition when the camera is occluded after an initial observation and before the contact command. To the best of our knowledge, this is the first work to extend recurrent gated fusion of onboard depth and proprioception from perceptive locomotion to humanoid loco-manipulation under interrupted visual sensing.