AI summaryⓘ
The authors created MatPhaseBench, a special test set to see how well AI models can understand complex phase diagrams used in materials science. These diagrams show how materials change with temperature and composition, which is hard even for computers because it needs deep scientific reasoning. They collected 200 carefully chosen diagram and text pairs from many scientific papers, making sure the descriptions are accurate and detailed. Their experiments found that current AI models can only recognize simple visual features and struggle with deeper understanding and detailed comparisons. This benchmark aims to help improve AI that can really understand complicated scientific images like phase diagrams.
materials phase diagramsthermodynamicsphase stabilityscientific image understandingvisual language modelssemantic alignmentmulti-modal AIbenchmarkmaterials science
Authors
Hanwen Wang, Sihan Liang, Zhiwei Liu, Yangang Wang, Wei Yan, Yuqin Liu, Zongguo Wang
Abstract
Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning. Although VLMs have shown promise in scientific image understanding, their systematic evaluation on such logically complex images demanding deep mechanistic interpretation remains limited, and phase diagrams provide a challenging testbed for this purpose. We introduce MatPhaseBench, a high-quality, high-reliability benchmark for complex scientific image understanding, focused on materials phase diagrams. MatPhaseBench is constructed from 3681 papers in classical materials science journals, from which 200 high-quality diagram-text pairs were selected, covering 189 material systems and 70 elements. The benchmark has three key features: (1)targeting complex scientific image understanding-it moves beyond simple objective tests to open-ended tasks requiring deep comprehension; (2)comprehensive image-text alignment-semantic information associated with images is fully preserved during literature mining and matching; (3) high-quality human-supervised text acquisition-all descriptions undergo strict manual validation. Experimental results show that current VLMs remain substantially behind expert-level understanding: they are largely limited to surface visual perception, lack deep reasoning grounded in thermodynamic mechanisms, have limited domain awareness and expert analytical experience, and perform poorly in distinguishing fine-grained differences in composite or multi-diagram settings. Overall, MatPhaseBench constitutes a challenging research-grade benchmark, providing a foundational platform for complex scientific image understanding, phase diagram analysis, and trustworthy multi-modal AI in science.