Robot learning benchmark measures transfer from human videos
H2RBench: A Real-to-Sim Benchmark for Evaluating Human-to-Robot Transfer
Robotics
Summary
Getting robots to learn tasks by watching humans can help teach them faster and more flexibly. But comparing ways to transfer human actions to robots is tricky because studies use different tasks and settings. The authors created H2RBench, a standard test that uses real human videos and robot simulations on four tasks to fairly compare these methods. They found some methods make better use of more human examples than others, and how well a method works in simulation closely matches real robot results. This benchmark helps developers pick and improve ways to teach robots by watching humans before trying them on real machines.
What this means in practice
- •For robotics developers: Compare and improve robot manipulation methods that learn from human videos using a standardized benchmark with simulation and real-world validation.
- •For automation engineers: Assess different approaches to robot teaching from human actions to deploy manipulation tasks more reliably before real robot testing.
Authors
Chuyang Xiao, Haotian Zhan, Sriram Krishna, Peilin Meng, Muhammad Zubair Irshad, Sergey Zakharov, David Held
Abstract
Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing different human-to-robot (H2R) transfer methods remains challenging, as existing approaches are evaluated under different settings, including differing task suites, scene layouts, object instances, and amounts of robot supervision. To address this challenge, we present H2RBench, a Real2Sim benchmark for evaluating H2R transfer methods. H2RBench provides a standardized protocol built on real human video demonstrations and simulated robot demonstrations, and includes four manipulation tasks spanning diverse interaction requirements. We evaluate multiple representative H2R transfer methods, each adopting a different strategy for bridging the embodiment gap. Using H2RBench, we systematically characterize how each method scales with the amount of human demonstrations, revealing that methods differ substantially in their ability to leverage additional human data. We further show that simulation performance is broadly predictive of real-world robot performance, with an overall Pearson correlation of r = 0.89, Spearman correlation of \r{ho} = 0.85 and Mean Maximum Rank Violation (MMRV) of 0.06 across method-task configurations. These results establish H2RBench as a practical and scalable benchmark for comparative H2R evaluation prior to real-world deployment.