Dataset releases 3,718 robot episodes with real-world task outcomes
Rollout then Recycle: 3,718 Labelled Real-Robot Episodes from Benchmarking 7 Policies on 12 Tasks
Robotics
Summary
Teaching robots to do tasks like picking up objects can be hard, and machines often fail. The authors collected over 3,700 episodes showing robot attempts on different tabletop tasks using various control methods. Many attempts failed, and each was carefully labeled as success, partial success, or failure. The dataset includes video views and robot actions, helping others study what works and what doesn't in real robot learning.
What this means in practice
- •For robotics developers: Use labeled real-world robot episodes to improve task learning and failure handling on low-cost robotic arms.
- •For industrial automation teams: Benchmark different robot control policies on shared tasks to select or fine-tune better automation solutions for tabletop operations.
Authors
Praveen Selvaraj, Lorenzo Uttini, Ville Kuosmanen
Abstract
We release all 3,718 episodes from the ArmnetBench v0.1 evaluation (arXiv:2607.24481), which contains 600 teleoperated demonstrations and 3,118 rollouts of seven imitation-learning and vision-language-action policies on 12 tabletop tasks, recorded on low-cost SO-101 arms; 73% of the rollouts are failures. Every policy for a task was trained or fine-tuned on the same 50 demonstrations and rolled out on the same cell, so the corpus contains many policies' failures on shared tasks. An operator labelled every rollout successful, suboptimal or failure. Each episode has three camera views, joint states and actions and is available on the Hugging Face Hub in LeRobot v3.0 and a reward-model format. This note documents how the episodes were collected and the limitations that could affect their use.