RoboValue benchmark tests robot task value judgments in real and simulated settings
RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models
Robotics
Summary
Robotic value models help robots understand how well they are doing on tasks by looking at what they see and instructions they get. Current tests don’t check all parts of a robot’s understanding well enough. The authors created RoboValue, a new tool that checks how well robots judge progress, handle failures, and stay consistent while doing tasks, both in simulations and real life. They found many mistakes in how current models track task steps and recover from errors. Their dataset and leaderboard help others improve robot feedback reliability.
What this means in practice
- •For robotics developers: Evaluate and improve robot software to better judge task progress and handle failures during real or simulated operations.
- •For industrial automation teams: Use RoboValue to test robotic arms and manipulators for more reliable feedback when performing assembly or handling tasks.
Authors
Shengbang Liu, Zhengye Du, Zhilong Wan, Honghao Su, Chenxiang Xia, Chang Ge, Jinyang Xiao, Nan Wang, Zhipeng Hu, Huining Yuan, Shu'ang Yu, Yue Chen, Tianxing Chen, Chuankang Li, Maoqing Yao, Wenbo Ding, Yu Wang, Guanghui Ren, Chao Yu
Abstract
Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations cover only selected aspects of execution understanding, leaving the reliability of value feedback insufficiently tested. We introduce RoboValue, a fine-grained sim-and-real benchmark for unified evaluation of robotic value models. Its framework evaluates four complementary capability dimensions through shared interfaces: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency. Diagnostic trajectories and instruction variations test whether value judgments accurately reflect task requirements and changes throughout execution. RoboValue-Dataset covers 15 simulation and 20 real-world manipulation tasks, providing expert demonstrations for training and annotated trajectories for fine-grained evaluation. We evaluate representative robotic value models in zero-shot and one-shot settings under standard conditions and separate shifts in robot embodiment and environment. Our results reveal errors in fine-grained instruction grounding, history-dependent progress tracking, and recovery assessment, even in models with strong outcome and forward-progress scores. The same semantic subtask can also receive substantially different value gains across valid execution orders. These findings identify concrete targets for value supervision, while the dataset and public leaderboard support continued assessment of feedback reliability. The website is available at https://robovalue-benchmark.github.io/.