Benchmark tests AI models on physics reasoning and discovery tasks
PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research
Computation and Language
Summary
Understanding and extending theoretical physics is hard, especially for AI. The authors created PHYRD, a set of physics problems that test if AI can reproduce known results and explore new theory variations. They show that current AI models have mixed success and make mistakes in reasoning. They also introduce FormulaForest, a tool to generate helpful training data that improves AI performance on related science tasks.
What this means in practice
- •For ai developers: Evaluate and improve AI models' ability to handle complex physics reasoning with a comprehensive benchmark and synthetic training data.
- •For scientific software engineers: Incorporate automated reasoning benchmarks like PHYRD to enhance tools for verifying and extending theoretical physics calculations.
Authors
Qituan Zhang, Guo Chen, Linfeng Zhang, Wenlong Zhang
Abstract
Large language models and agents have shown significant power in advanced physical research. In this paper, We introduce PHYRD, a living benchmark for theoretical-physics reasoning with 63 Reproduce and 43 Discover problems. Reproduce tests derivation of published results; Discover tests consequences of modified theoretical conditions. FormulaForest constructs the tasks with reference solutions and subquestion-level rubrics. Across five models, whole-problem accuracy ranges from 49.21% to 68.25% on Reproduce and from 18.60% to 74.42% on closed-book Discover. With tools, four models complete more Discover problems, but conceptual errors and missing arguments persist. Besides, our experimental results demonstrate the value of FormulaForest as synthetic data. On FrontierScience, a benchmark developed by OpenAI, it raises the accuracy of a Qwen-35B-A3B model from 61% to 72%.