Variational automation benchmark measures robot skill across tasks
VABAR: A Variational Automation Benchmark for Agentic Robotics
Robotics
Summary
Robots in factories and hospitals need to do specific jobs even when the tools or parts change shape or position. The authors call this challenge Variational Automation and they designed a new benchmark called VABAR to test how well robots can handle these changing tasks. VABAR uses simulations of six different real-world tasks, like sorting tools or inserting cables, and includes lots of variations and difficulty levels. It provides open-source code and examples so robot builders can check how good their robot control systems are. The authors also showed initial results using some existing robot control agents.
What this means in practice
- •For industrial robotics engineers: Evaluate robot control systems on variable industrial tasks like gear meshing and tool hanging using a standard, open simulation benchmark.
- •For medical device developers: Test robotic manipulation in medical tasks such as syringe packing with a repeatable simulated benchmark featuring realistic variations.
Authors
Shuangyu Xie, Kaiyuan Chen, Uksang Yoo, Xinyue Zhu, Meadow Shen, Jaimyn Drake, Eugen Solowjow, Lionel Gulich, Alexander Millane, Rafael Cathomen, Tuur Stuyck, Sangeeta Subramanian, Yichao Pan, Bruce McGowan, Michael Benedict, Linxi Fan, Yuke Zhu, Spencer Huang, Ken Goldberg
Abstract
For robots to operate reliably in commercial and industrial applications, they must persistently perform specific tasks with known variations in object shape and pose. We call this Variational Automation (VA) to differentiate these settings from fixed automation. Inspired by recent papers on Agentic Robotics, where pretrained LLM and VLM coding agents combine interpretable robot programming with the open-world adaptability of model free policies, we propose a general set of agentic robotics benchmark design criteria that includes accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency. We instantiate these principles in six manipulation task classes spanning industrial, medical, and infrastructure applications: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. We introduce the Variational Automation Benchmark for Agentic Robotics (VABAR), an open, easily replicable, simulation-based manipulation benchmark to systematically evaluating agentic robotics systems. Built fully on open-source tools, VABAR features detailed implementations using the Newton physics engine, standardized task variations, difficulty tiers, success metrics, and a library of 33 generalizable skills with human-authored programmatic reference solutions. We provide initial baseline evaluations using an open agentic manipulation harness, Graph-as-Policy (GaP), and Isaac Lab Arena.