Humanoid robots retrieve past depth data for safer foothold prediction
FootQuery: Future-Touchdown-Guided Retrieval from Depth History for Perceptive Humanoid Locomotion
RoboticsMachine Learning
Summary
Walking robots need to know where to put their feet next, especially when the ground ahead is hard to see. This paper introduces a system called FootQuery that helps robots remember past camera views of the ground and use predicted foot positions to find good spots to step. The authors trained their system so the robot matches its foot contacts with earlier images to learn where it can safely step. Their approach helps robots climb stairs and cross gaps more smoothly, tested in both simulations and real-world settings.
What this means in practice
- •For robotics engineers: Enable humanoid robots to anticipate next foot placements using past depth images for stable walking on complex terrain such as stairs and gaps.
- •For robotics software developers: Incorporate historical depth data querying based on predicted foot touchdowns into robot control policies to improve environmental awareness during locomotion.
Authors
Tao Dong, Jia Yu, Yuxuan Fan, Linna Zhao, Jiaqi Gong, Andong Yang, Chao Gao, Guyue Zhou
Abstract
Humanoid locomotion over complex terrain requires anticipating footholds that may no longer be visible at touchdown. Limited camera coverage and self-occlusion make it necessary to retrieve relevant terrain information from earlier observations. We present FootQuery, a perceptive locomotion framework that queries depth history using each foot's predicted next touchdown. The policy predicts touchdown locations and uncertainty from proprioception and uses these distributions, together with per-foot features, to query sparsely sampled historical depth frames. During training, realized contacts are projected into historical images to supervise retrieval at the regions where those contacts were visible. The retrieved per-foot features are fused with global visual memory to generate control actions. A progressive force-assistance curriculum supports early exploration, while event-consistent tread-midline shaping encourages coordinated stair contacts. Deployment requires only proprioception and onboard depth images. In simulation, the complete framework outperforms its component ablations on the most challenging tested stairs, gaps, and platforms. Real-world experiments on a Unitree G1 demonstrate continuous traversal with a single policy across outdoor stairs and indoor routes combining stair ascent and descent, platforms, and gaps. These results support organizing visual history around anticipated contacts for perceptive humanoid locomotion.