Benchmark assesses robot learning manipulation skills from videos

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation

Robotics

Summary

Robots can learn how to do tasks by watching videos of humans, similar to how people learn from each other. But it has been hard to compare different approaches because they use different setups and assumptions. The authors created RoboReel, a standard test set with videos, robot actions, and tasks to fairly compare methods for robot learning from human demonstrations. They tested several state-of-the-art algorithms and found that current models still struggle with complex tasks that require many steps or precise accuracy.

What this means in practice

  • For robotics engineers: Evaluate and compare robot learning algorithms using a shared benchmark with real human videos and simulated tasks.
  • For automation software developers: Develop better robot control systems by testing new methods on RoboReel’s comprehensive and diverse manipulation tasks.

Authors

Weiwei Gu, Anmol Gupta, Anant Sah, Ryan Varghese, Lalitha Shreya Vanam, Prabhath Adireddi, Peter Karkus, Nakul Gopalan

Abstract

Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its biological parallels, this modality provides a practical solution for data scaling in sample-inefficient and data-starved domains like robotics. Recent work has demonstrated promising results in learning manipulation skills from human videos, yet progress in this area remains difficult to assess. Existing methods vary widely in assumptions, hardware choices, and environment setups making it difficult to draw meaningful comparisons and identify advances in the field. To address these challenges, we introduce RoboReel: a unified benchmark for evaluating models that learn policies from human videos. RoboReel consists of bundled real-world human demonstration videos, simulated robot trajectories, and evaluation environments on ten manipulation tasks. We develop four test suites to evaluate the models' performance on multiple axes, including the robustness to visual distractors and the ability to complete long-horizon tasks. Our benchmark covers learning-from-observation models from different categories, and studies the effectiveness of multiple representation choices in our benchmark evaluation that covers over seven state-of-the-art algorithms (including our VLA based variants) in the field of LfO. Finally, we present an analysis of the different types of algorithms showing that long-horizon tasks and tasks with low tolerances are still challenging for current models. Webpage: https://roboreel.github.io