Multimodal benchmark measures AI skill in lab decision making

LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making

Artificial Intelligence

Summary

Laboratory experiments don't just follow fixed steps; they need careful choices based on changing evidence. The authors created LabOPBench, a test with real lab questions requiring these decisions and combining different types of information. They found current AI models still struggle, scoring low compared to expert humans. The biggest problems are misunderstanding evidence or conflicting it with decisions. This shows AI has a long way to go before it can reliably help in real lab work.

What this means in practice

Authors

Jingyuan Li, Yujie Liu, Yijie Dai, Xiaoyu Yuan, Nian Ran, Huajie Niu, Ming Zhu, Shuwen Zhao, Chenxi Li, Fenghao Lu, Jiahao Li, Xinyu Gu, Zhe Liu, Long Xu, Wenlong Zhang, Bo Zhang, Lei Bai, Peiyu Zhang, Minting Pan, Weinan Zhang, Dongzhan Zhou

Abstract

Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.