RxnExBench benchmarks AI for extracting chemical reaction data
RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature
Artificial Intelligence
Summary
Scientific papers about chemistry often have complex tables and pictures showing chemical reactions, which are hard for computers to understand automatically. The authors created RxnExBench, a large collection of carefully labeled chemical reactions and molecules to test how well computer models can read and understand these figures. Their tests show that current AI models struggle especially when the chemistry details get complicated or need deep reasoning. They also found that just recognizing molecules is not enough; the AI needs to combine images and text and understand chemistry to do better.
What this means in practice
- •For chemical data engineers: Benchmark vision-language models to improve automated extraction of reaction information from chemistry literature tables and figures.
- •For pharmaceutical informatics teams: Enhance retrosynthetic planning and reaction condition prediction tools by integrating fine-grained chemical reaction data extracted via RxnExBench.
Authors
Jiaxing Sun, Junyuan Gao, Ruikang Zhang, Hao Zheng, Lingli Ge, Xiao An, Jiang Wu, Huazheng Zeng, Jiahe Song, Chen Zhu, Haote Yang, Jingchao Wang, Lu Chen, Yuanyuan Cao, Ruijie Zhang, Shengxin Lu, Weijia Li, Lijun Wu, Conghui He
Abstract
Screen Tables and Substrate Scopes are information-dense figures widely used in synthetic chemistry literature, yet automated extraction of structured reaction data from them remains challenging. We introduce RxnExBench, a multimodal benchmark for fine-grained reaction-level evaluation. Built from high-impact chemistry journals, it contains 3,565 expert-annotated reactions and 2,361 molecular structures. The dataset adopts a unified schema with 15 reaction roles and 26 core fields, supporting downstream AI-for-Chemistry tasks such as retrosynthetic planning, yield prediction, and reaction condition recommendation. RxnExBench further provides a reasoning-driven difficulty taxonomy, a modular annotation pipeline, and the Oracle-MolRec protocol, which disentangles molecular recognition from higher-level reasoning. Evaluation of 9 state-of-the-art VLMs shows that performance degrades substantially as schema complexity and reasoning demands increase. Under the Oracle-MolRec setting, results further reveal that accurate molecular recognition alone is insufficient, with multimodal integration and chemistry-aware reasoning remaining the primary bottlenecks. We release all data and code to facilitate future research in chemistry-aware multimodal understanding and AI-for-Chemistry dataset construction. The dataset and code will be released at https://github.com/opendatalab/RxnExBench.