Human object motions guide robots with different bodies
Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
RoboticsArtificial Intelligence
Summary
It is hard to teach robots by watching humans because robots often look and move differently than people. The authors show a way to record how an object moves during a human demonstration, instead of tracking the person's hand. This recording can then help different robots figure out how to move the object using their own unique bodies. They tested this method with several robots and many recordings, showing it can tell when a robot can or cannot perform the task just from simulation.
What this means in practice
- •For industrial robot integrators: Use recordings of human object movements to program diverse robots without retraining on each robot’s hand motions.
- •For medical device developers: Adapt human demonstrations of object manipulation for surgical robots with different mechanical structures to improve training efficiency.
Authors
Nagasrikanth Kallakuri
Abstract
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.