Language models tested on full molecular cloning experiment plans
BenchBench-Cloning: Evaluating Molecular Cloning Design and Protocol Development
Artificial Intelligence
Summary
Molecular cloning is a common lab process where biologists create specific DNA sequences for experiments. The authors made a benchmark called BenchBench-Cloning to test how well language models can plan entire cloning experiments, including designing DNA constructs and writing detailed lab protocols. They found that even top models struggle to successfully plan full cloning workflows, with the best one achieving just over 30% success. This shows how complex experimental planning is and gives a new way to measure progress in AI tools for biology.
What this means in practice
- •For biotech lab automation teams: Generate executable molecular cloning protocols automatically for specified DNA constructs to streamline experiment setup and reduce manual planning errors.
- •For synthetic biology developers: Design DNA constructs aligned with scientific objectives to accelerate experimental design in synthetic biology projects.
Authors
Aditya Sivakumar, Peter Ciaccia, Hassaan Qadir, Ian Hsu, Ashu Singhal
Abstract
Molecular cloning is one of the most routine wet-lab workflows, yet existing evaluations of language models in biology mostly test knowledge recall or isolated steps rather than end-to-end experimental planning. We introduce BenchBench-Cloning, a benchmark for large language models comprising 65 molecular cloning tasks grounded in sequence records, laboratory notebooks, and published experimental studies. Planning a molecular cloning experiment requires defining a set of constructs needed for a scientific objective and a construction route to produce them. BenchBench-Cloning contains 50 protocol-development tasks, in which agents produce an executable protocol for a specified construct, and 15 construct-design tasks, in which agents determine the constructs a scientific objective calls for. Protocol-development tasks are evaluated by execution in a cloning simulator; construct-design tasks are evaluated with task-specific deterministic checks and, where needed, rubric-based LLM-as-a-judge evaluation. Across eight systems with repeated attempts per task, GPT-6 Astra scores highest at 30.6%, with the others between 8.6% and 22.2%, and the benchmark remains unsaturated under best-of-ten sampling. As integrating individual cloning decisions into complete, valid plans remains challenging even for frontier models, we expect BenchBench-Cloning to serve as a useful benchmark for tracking progress in experimental planning.