Summary
Teaching humanoid robots to interact with objects and environments by watching videos of humans is challenging because of differences in body shapes and how robots understand contact with the world. The authors present VICAR, a system that carefully adapts human motions from videos to humanoid robots while keeping important contact points and timing intact, so the robot can perform tasks like climbing ladders or picking up objects smoothly. Their method improves how robots learn from videos by focusing on preserving physical contact details and adjusting motions for real-world changes. This approach helps robots learn practical interactive behaviors by simply observing humans.
What this means in practice
- •For robotics engineers: Develop humanoid robots that learn complex interactive tasks from third-person human videos using contact-aware motion adaptation.
- •For interactive entertainment developers: Create realistic humanoid characters that replicate human object interactions and movements in video games or simulations by retargeting human video data.$Commercial implications: Enables development of lifelike humanoid avatars for entertainment by automating motion adaptation from human videos.
Abstract
Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. Recent video-to-humanoid systems have shown that noisy human motion, object trajectories, and contacts can be extracted from unstructured videos and refined into deployable skills. However, a key gap remains for contact-rich imitation: the task is often defined by world-frame contacts whose poses, timing, and speed must survive embodiment mismatch and runtime scene changes. Towards this, we present VICAR (Video Imitation with Contact-Aware Retargeting), a pipeline for learning humanoid interactive skills from third-person human videos. The system first recovers human motion, reconstructs task-relevant obstacles from monocular video, and then retargets the motion to a humanoid while preserving global contact poses, motion speed, and collision clearance. Unlike retargeting methods that only remove penetration or preserve local interaction geometry, our formulation represents user-specified contact poses and collision clearance as high-priority Lagrangian penalty terms and optimizes the full motion to remain smooth through contact-rich transitions. The resulting motions are used to train a contact-conditioned motion generator whose inputs include movable contact points enabling runtime adaptation to new contact positions. We demonstrate the framework on ten diverse third-person video skills, including six table-tennis serves, three object-pickup tasks, and ladder climbing, and deploy the learned skills on hardware. These results suggest that contact-preserving retargeting is a practical step toward humanoids that learn useful interactive behavior by watching humans.