Jointly retarget human and object motion for humanoid robots
OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport
Robotics
Summary
Turning human actions into movements a robot can perform is tricky, especially when the robot needs to interact with objects and the environment just like a human would. The authors created a method that copies both the robot’s body movements and the objects being moved simultaneously, keeping interactions accurate despite differences in size and shape. Their approach uses math to match surfaces and directions between humans, robots, and objects, then solves how the robot and objects should move together at every moment. Tests show this method works better than a previous one and can control a real robot doing tasks like moving boxes with both hands.
What this means in practice
- •For robotics engineers: Generate accurate humanoid robot motions that maintain environment and object interactions from human demonstrations for manipulation tasks.
- •For robotic automation teams: Control humanoid robots in warehouse or manufacturing settings to handle objects using whole-body coordinated motions transferred from human operators.$Commercial implications: Enables producing robotic systems that automate picking and placing tasks using human-like coordination, improving efficiency and versatility.
Authors
Guillaume Besset, Erwann Carn, Timothée Carecchio, Valentin Tordjman-Levavasseur, Fabian Schramm, Yann de Mont-Marin, Justin Carpentier, Ajay Suresha Sathya
Abstract
Transferring human motion to humanoid robots requires adapting the demonstrated motion to the robot morphology while preserving interactions with the environment. This is particularly challenging for loco-manipulation tasks, where contacts with the ground and manipulated objects must remain consistent despite differences in body proportions. Yet, skeletal motion alone does not fully describe these interactions, and fixing object trajectories limits the adaptation to a new embodiment. In this paper, we introduce OTR ETARGET, a unified approach to jointly retarget robot and multi-object motion from human demonstrations. Our approach represents surface interactions through signed distances, closest surface points, and relative directions, and uses entropic optimal transport to transfer these quantities across human, robot, and object geometries. We incorporate the resulting interaction targets into a constrained inverse kinematics formulation that balances contact preservation with motion style and jointly optimizes robot and object poses at each frame. This formulation accommodates robot-object and object-object interactions without rescaling the scene or the demonstration. We validate the proposed approach on OMOMO, where it achieves a robot- object interaction Jaccard score of 87% and a depth error of 8.7 mm, compared with 28% and 29.3 mm for OmniRetarget. Finally, we demonstrate transfer to a physical G1 humanoid using whole-body policies trained with reinforcement learning on the retargeted references, across motions including two-handed box pick-and-place onto a table.