Crystal structure prediction gains new comprehensive evaluation benchmark

Quantitative Assessment of Crystal Structure Prediction

Machine LearningArtificial Intelligence

Summary

Predicting the 3D shape of crystals from their chemical makeup is important for developing new materials. Existing evaluation methods for this prediction either rely on hard-to-compare metrics or ignore whether the predicted structures physically make sense. The authors created a new benchmark called OPAL-Bench to fairly evaluate prediction methods, considering multiple stable crystal forms and physical realism. They also provide a new baseline method that performs well on these combined measures. This benchmark helps developers better judge and improve crystal structure prediction tools.

What this means in practice

  • For material scientists: Evaluate and compare crystal prediction models to improve material design for applications like energy storage and biomedical devices.
  • For pharmaceutical developers: Assess multiple stable crystal structures of drug compounds to optimize formulation and synthesis processes.

Authors

Sergio Rincón, Gabriel González, Nicolás Andrade, Rafael Velásquez, Sebastián Ojeda, Juanita Puentes, Nicolás Aparicio, Paula Cárdenas, Pablo Arbeláez

Abstract

Crystal Structure Prediction (CSP) aims to predict a material's 3D structure from its chemical composition, with applications from energy storage to biomedical devices. Despite its importance, existing evaluation frameworks suffer from key limitations: they rely on threshold-sensitive or incomparable metrics, focus predominantly on geometric performance while overlooking physical plausibility, and fail to account for polymorphism, where a chemical composition can crystallize into multiple stable structures. In parallel, recent Symmetry-Informed CSP (SICSP) methods often report results alongside standard CSP baselines while receiving crystallographic templates at inference, which constrain the search space and make SICSP a fundamentally different task. To address these limitations, we introduce an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) that evaluates CSP and SICSP as separate tasks. We evaluate 7 CSP and 2 SICSP methods under OPAL-Bench across 4 datasets using threshold-free geometric and physical distance metrics. Our results show that no method dominates across all metrics, and that geometric performance is not strongly correlated with physical plausibility. We also release OPAL-CSP, an SE(3)-equivariant flow-matching baseline for the CSP track that performs competitively on geometric metrics and ranks among the best on physical ones. By unifying geometric and physical metrics under a polymorph-aware protocol, OPAL-Bench enables a more comprehensive assessment of generative quality in CSP.