MatToolBench measures AI agents in real materials science software
MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows
Artificial Intelligence
Summary
Multimodal AI agents can do many tasks in regular software, but it's unclear how well they work with highly specialized scientific tools. The authors created MatToolBench, a set of 204 real-world tasks in materials science software, to see how well these AI agents perform. Their tests showed that current models struggle with these professional workflows, succeeding in only a quarter to half of tasks. The difficulties come from missing specialized knowledge and complex software interactions that general training doesn't cover.
What this means in practice
- •For materials science software developers: Evaluate and improve AI assistants that operate complex scientific software for materials research using real workflow tasks.
- •For automation engineers in scientific computing: Test multimodal AI agents in handling integrated GUI and scripting tasks to automate materials science workflows.
Authors
Mei Wu, Rui Xie, Runyu Zhang, Yuqiang Li, Tianfan Fu, Bo Chen, Kai Yu, Xin Chen, Lu Chen
Abstract
Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software remains largely unexplored. In materials science, sparse domain-specific web data, specialized interfaces, and tacit workflow conventions create blind spots that general-purpose pretraining cannot readily bridge. We present MatToolBench, the first real-environment benchmark for evaluating multimodal GUI agents on professional materials science software, comprising 204 tasks across 10 tools in three modalities: GUI operation, OriginPro scripting, and code-based database queries, all executed inside a Windows 11 VM. Each task is decomposed into fine-grained sub-criteria by domain experts, enabling interpretable partial-credit scoring; the GUI component of our multi-level evaluation pipeline achieves an average F1 of 0.98. For OriginPro figure-generation tasks, we further conduct a human-LLM agreement study to validate the use of a multimodal judge for secondary aesthetic assessment. Our experiments show that strong performance on general benchmarks does not transfer to professional scientific workflows, and that this gap is not a visual-grounding problem alone: failures arise from domain-specific operational knowledge, sparse pretraining coverage of scientific software, weak cross-tool artifact handoff, and critical states exposed only visually. Even the best model reaches only 25% success rate on GUI tasks and 45% on code tasks. MatToolBench therefore serves as a challenging diagnostic benchmark and real-environment testbed for data-scarce, knowledge-intensive scientific workflows.