Summary
Humanoid robots usually follow pre-recorded motions but struggle to react to changing environments because they lack visual feedback in their control systems. The authors present ViBe, a way to add vision-based sensing to existing motion trackers by using pre-trained visual encoders and fine-tuning them efficiently for specific tasks. This enables robots to better handle activities like walking over curbs, parkour, object manipulation, and even playing dodgeball, while transferring these skills from simulation to the real world. The approach was tested in various conditions including outdoors and low light, showing robust performance without needing a complex planner.
What this means in practice
- •For robotics engineers: Enhance humanoid robots to adapt motions using vision feedback for complex whole-body tasks in real-world settings.
- •For industrial automation teams: Implement perceptive controls in robots to improve interaction with diverse objects and environments, like handling objects while moving.
Authors
Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen
Abstract
Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.