MLIP Detective finds hidden errors in machine-learning atomic models
MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials
Machine Learning
Summary
Machine-learning interatomic potentials (MLIPs) are tools used to predict how atoms behave, but sometimes they fail in ways not visible in standard tests. The authors created MLIP Detective, which uses physics knowledge to actively search for and identify hidden mistakes in these models. For example, it found that a popular model incorrectly predicted certain molecules on surfaces to be less stable than they should be. MLIP Detective can help improve MLIPs by pointing out such errors and suggesting ways to verify them.
What this means in practice
- •For materials simulation teams: Detect hidden failure modes in machine-learned atomic interaction models to improve simulation reliability.
- •For computational chemistry developers: Screen and verify machine-learning potentials for flaws before integration into chemical modeling pipelines.
Authors
Ryuhei Okuno, Nontawat Charoenphakdee, Kaoru Hisama, Yuta Tsuboi
Abstract
Universal machine-learning interatomic potentials (u-MLIPs) aim to generalize across diverse configurations. Benchmarks enable reproducible evaluation but may not expose failures outside their predefined scope. Here, we show that physics-informed search can complement benchmark-based evaluation by uncovering hidden failure modes. We introduce MLIP Detective, an agentic framework for active failure mode discovery. Starting from benchmark evidence, MLIP Detective generates falsifiable, physics-informed failure hypotheses, screens them with inexpensive simulations, and escalates only the most suspicious cases to human experts together with proposed verification protocols. Without issue-specific prompting, MLIP Detective identified and characterized a systematic anomaly in MACE-MPA-0: the model predicted some relaxed adsorbate-surface systems involving O- or F-containing adsorbates to be higher in energy than their corresponding separated fragments. Using cross-model comparisons, MLIP Detective further inferred a likely training-data origin for the anomaly, consistent with recent reports.