Robot benchmark measures how lab actions respond to real outcomes
Labference: Benchmarking the Reasoning-Action Connection in Robotic Laboratories
Robotics
Summary
Robots in scientific labs need to adjust their next steps based on what actually happens during experiments, like moving things correctly or dealing with mistakes. The authors created Labference, a test that checks how well robots observe and respond to these physical results when choosing their next actions. They found that some robots make good choices but fail to execute actions properly, while others struggle with selecting the right target despite moving objects successfully. This new benchmark helps identify where robots need to improve so they can do experiments more autonomously and safely.
What this means in practice
- •For robotics engineers: Use Labference to evaluate and improve how robots adjust their lab procedures based on real execution outcomes.
- •For industrial automation teams: Apply Labference to diagnose when automated systems fail to connect reasoning with physical actions in manufacturing tasks.
Authors
Shitong Liu, Jingwei Zhang, Sheng Lian, Xinyu Zhuang, Kaiwen Guo, Quan Liu, Chunhai Zhao, Jiaxin He, Fan Yang, Manli Tao, Tingyu Yuan, Jinqiao Wang, Hongwei Shi, Chaomin Guo, Xiao-jing Liu, Xiangyu Zhu, Zhuang Xiong, Feichan Yang, Chaoyang Zhao, Zhiyong Liu
Abstract
Realizing Embodied AI for Science requires robots to translate experimental goals into reliable physical procedures, reducing hazardous and repetitive human work while generating evidence for scientific discovery. Although language-based planning and learned manipulation provide complementary capabilities, their integration requires robots to adapt subsequent operations to actual execution outcomes. For example, an unsuccessful placement changes what must be done next, even when the experimental goal remains unchanged. Existing benchmarks assess planning, manipulation, and laboratory workflows through task completion and reasoning or execution metrics; however, aggregate performance leaves the connection between outcome-dependent choices and physical execution unresolved. To address this evaluation gap, we introduce Labference, a benchmark for evaluating how observed physical outcomes guide subsequent choices and how those choices translate into robotic laboratory actions. Its evaluation protocol is designed to test outcome-dependent behavioral adaptation through controlled evidence variations, while independently measuring operation selection, physical execution, and task completion. Matched Oracle-assisted continuations from shared checkpoints then assess which forms of support improve subsequent choices and task completion. Initial weighing evaluations reveal target-selection errors in the tested VLA policies despite successful object transfers, whereas GPT-driven control achieves correct selection but encounters execution failures and substantial computation costs. Together, these findings highlight the need to connect task-appropriate reasoning with reliable, timely action. By making this connection measurable and diagnosable, Labference supports targeted improvements toward autonomous experimentation for Embodied AI for Science.