SciResearch-Eval benchmarks AI scientific judgment with flawed simulators

SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

Artificial IntelligenceMachine Learning

Summary

Measuring how well AI can design experiments and judge scientific data is hard, especially when the tools used have errors. The authors created SciResearch-Eval, a test where AI agents try to find material heat properties by designing experiments while using imperfect computer simulations. They found most models are often overconfident and wrong when simulations miss physics, but some advanced AIs manage better by choosing experiments that work well despite flaws. This helps compare how well AIs can gauge uncertain science problems and suggests better ways to calibrate their confidence.

What this means in practice

  • For ai developers: Evaluate and improve AI models' ability to design experiments and assess uncertainty using interactive flawed simulations.
  • For industrial r&d teams: Test AI-driven experimental design tools on realistic but imperfect simulators to better understand confidence in industrial measurements.

Authors

Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora

Abstract

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.