Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
2026-09-01 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors created SCILAWS-BENCH, a new set of scientific problems to test if AI models can discover scientific laws from real data and published research. Their benchmark includes 118 problems across six scientific fields and tests models in two ways: using fixed observations to find laws and actively exploring to uncover hidden ones. They found that good prediction does not always mean the discovered law is scientifically valid, and that models sometimes just memorize existing formulas instead of finding new ones. Their work helps better evaluate AI’s abilities in scientific discovery using real-world challenges.
scientific law discoverylarge language modelsbenchmarkpredictive fitscientific validitymemorizationactive learningreal-world dataAI for Sciencehypothesis testing
Authors
Yiming Huang, Ziche Liu, Zhuohang Wu, Yiqian Wang, Junxia Cui, Xinkai Zou, Linjun Mao, Nan Huang, Naicheng Yu, Kaijie Zhu, Yue Ma, Kun Zhou, Letian Peng, Jingbo Shang
Abstract
Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: https://yiyihum.github.io/SciLaws-Bench