Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition
2026-08-10 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how well computer models can read chemical structure drawings from real documents compared to synthetic (computer-generated) images. They found that while models work very well on synthetic images, their accuracy drops a lot on real-world images. By fine-tuning these models with some real labeled images, the accuracy improved significantly, showing that real data is crucial. They also discovered that different base models and adaptation methods perform differently depending on the amount of real data available. These findings suggest that choosing and fine-tuning models should be tailored specifically for the real visual task at hand.
Optical Chemical Structure Recognition (OCSR)Synthetic imagesReal-world benchmarksFine-tuningVision language models (VLM)Vision-tower LoRAExact match accuracyHandwritten image recognitionBase model adaptation
Authors
Yani Guan, Dengpan Dong, Zi Wei, Shuang Luo, Dan Hannah, Yumin Zhang, Kang Xu
Abstract
Millions of chemical structures appear in patents and papers only as drawings, and using that information at scale requires reading the drawings. OCSR appears nearly solved on synthetic images yet remains difficult on real documents: the starting recognizer, Qwen2.5-VL-7B, exceeds 91% accuracy on synthetic renders but falls below 16% on three real-world benchmarks (ACS, CLEF-IP, USPTO). To identify the main source of improvement, 21 recognizers were fine-tuned on mixtures of synthetically rendered structures and labeled real depictions from patents, journal figures, and hand-drawn collections, varying the vision language model (VLM) base, the fraction of real training data, and the vision-tower adaptation strategy. Labeled real training images make the largest difference. For Qwen2.5-VL, ACS exact match rises from 0.15 with no real data to 0.37 at 9.5% and 0.46 at 50.2%; a controlled experiment across three base models reproduces the trend. A vision-tower LoRA, in contrast, does nothing for Qwen (+0.00, paired p=1.00), substantially helps InternVL3-8B (+22.8 to +34.6 pt), and modestly helps GLM-4.1V-9B (+1.0 to +9.6 pt), so its value depends on the base model. The best configuration reaches 0.96 exact match on clean renders and 0.49, 0.65, 0.84, and 0.76 on ACS, CLEF-IP, UOB, and USPTO, respectively. Gaps between base models are largest without real data (0.21), shrink to 0.06 at 70% real data, and reorder the ranking; base model and real-data mixture must therefore be selected together. Small-scale experiments on handwritten image-to-LaTeX recognition and chart-to-table conversion show that base-model rankings also vary beyond chemistry. More generally, model and adaptation choices for visual structure recognition should be evaluated on the target task.