System compiles science protocols into simulated robot tasks for testing
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Artificial Intelligence
Summary
Scientific experiments often require robots to carry out specific tasks, but turning written instructions into robot actions is hard and usually done by hand. The authors built SciHorizon-eLab, a system that automatically converts natural-language science protocols into tasks that robots can perform in a simulation. This system also checks that the tasks match the original instructions and can generate example runs and evaluations. They created a large benchmark of 300 tasks to help test how well robots can follow scientific experiment steps, finding that current methods often struggle to succeed.
What this means in practice
- •For robotics developers: Use the system to automatically convert lab experiment instructions into robot simulation tasks for developing and testing manipulation policies.
- •For automation engineers: Employ the benchmark to evaluate and improve robots aimed at laboratory automation with reliable step-by-step task performance measures.
Authors
Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu, Zirui Liu, Hongting Niu, Yuanchun Zhou, Hengshu Zhu
Abstract
Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.