Ai systems struggle to understand chemical placeholders in patents
R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing
Artificial IntelligenceMachine Learning
Summary
Understanding special chemical diagrams called Markush structures is hard for AI. These diagrams use placeholders to represent different molecular groups, common in drug patents. The authors created a new test called R-GroundBench to see how well AI can identify and work with these placeholders. They found current AI models do very well on easy tasks but struggle a lot with harder, more realistic ones. This shows that AI still has a long way to go before it can reliably help with chemical design tasks involving these complex structures.
What this means in practice
- •For pharmaceutical ai developers: Evaluate and improve AI models tasked with interpreting variable chemical groups in patent documents for drug design.
- •For chemical data annotation teams: Use the R-GroundBench benchmark to better assess and standardize the quality of annotations involving Markush structures.
Authors
Xin Wang, Zichuan Ying, Xinna Lin, Junqi Zhang, Hanyi Xiong, Tianyu Gao, Hairong Zhang, Qixiang Hua, Botian Shi, Zhenhailong Wang, Kaicheng Yu
Abstract
Recent advances in AI for scientific discovery enable molecular understandingand design, yet reasoning over incomplete chemical representations remainsunclear.Markush structures, which encode molecular families through variable R-groupplaceholders (\textit{R\textsubscript{1}}, \textit{R\textsubscript{2}}, \textit{X}, etc.), are ubiquitous in pharmaceutical patents and requiregrounding across molecular, textual, and chemical information.However, existing molecule-language benchmarks focus on fully specifiedmolecules, leaving R-group grounding largely unevaluated.We introduce R-GroundBench:, a diagnostic benchmark built from real patent Markushstructures, featuring a Multiple-Choice (VQA) track with controlled difficultyand modality splits, and an open-ended Generation track.Our results reveal a substantial gap between recognition andmolecular grounding.While models achieve over 90\% accuracy on Easy VQA, performance drops to56--66\% on Hard VQA when shortcuts are controlled.Chemical-domain VLMs also remain unreliable, achieving only 25.7--46.2\% on HardVQA despite domain-specific pretraining.Moreover, Generation Exact Match remains below 20\% for most models and below8\% when visual input is required.These findings reveal that current AI systems lack reliable grounding andexecution for Markush editing, highlighting challenges for AI-drivenscientific discovery.