AI agents evaluated on realistic tasks in materials science research
CompMat-Bench: Benchmarking AI Agents for Computational Materials Science
Artificial IntelligenceMachine Learning
Summary
Doing scientific research with AI is hard because experiments and simulations take a lot of time and computing power. The authors created CompMat-Bench, a set of 94 mini tasks based on real materials science studies, that tests AI agents on parts of research without needing costly simulations. They prepared the test tasks and known results ahead of time, so AI can be scored based on comparing outputs to these results. They found some AI systems do well with detailed instructions, but struggle with longer projects or less guidance. Most mistakes were from not understanding science rather than technical errors.
What this means in practice
- •For materials science software developers: Evaluate and compare AI tools designed to assist with material simulations without running expensive experiments during testing.
- •For machine learning engineers in scientific computing: Use a realistic benchmark to improve AI agents’ understanding of complex scientific workflows and troubleshooting their common failure points.
Authors
Chenmu Zhang, Levi Felix, Jun-Jie Zhang, Xingfu Li, Xuelian Jiang, Tao Jiang, Subhendu Mishra, Xixi Qin, Boris Yakobson
Abstract
Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical. We introduce CompMat-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66.0-90.4% across 94 tasks. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance. Failure analysis attributes most failures to scientific errors rather than to errors in software usage. CompMat-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes.