Papers for

industrial r&d teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

SciResearch-Eval benchmarks AI scientific judgment with flawed simulators

SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

Abstract: We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.

Mon 16 NovArtificial IntelligenceMachine Learning
The gist
Measuring how well AI can design experiments and judge scientific data is hard, especially when the tools used have errors. The authors created SciResearch-Eval, a test where AI agents try to find material heat properties by designing experiments while using imperfect computer simulations. They found most models are often overconfident and wrong when simulations miss physics, but some advanced AIs manage better by choosing experiments that work well despite flaws. This helps compare how well AIs can gauge uncertain science problems and suggests better ways to calibrate their confidence.
Open → 2611.21541v1

Open source toolkit speeds up intelligent experiment design research

TestBED: An Open-Source Toolkit for Bayesian Experimental Design

Abstract: Bayesian experimental design (BED) is a principled framework for intelligent data acquisition with applications across a wide range of scientific and industrial domains. However, the development and adoption of BED remains hindered by a lack of standardised software and benchmarks. Existing methods are typically implemented in bespoke codebases and evaluated on a small subset of tasks, making methods difficult to apply, compare, or extend. To address this, we present TestBED, a general-purpose, open-source toolkit for Bayesian experimental design. TestBED decomposes the BED process into modular, reusable components spanning tasks, objectives, training mechanisms, policy architectures, and evaluation procedures, together with scalable implementations of core primitives such as expected information gain estimators. We provide a wide range of these components, which can be freely composed, allowing existing methods to be reproduced within a common framework, while also enabling previously unexplored configurations and new methodologies to be easily implemented and evaluated. Using TestBED, we further conduct a large-scale benchmark of contemporary BED methods, including controlled studies of under-appreciated but surprisingly critical methodological choices, such as the importance of time embeddings in design policy architectures. By providing a modular, extensible software foundation and a reproducible benchmarking suite, we hope that TestBED will accelerate methodological research in BED and lower the barrier to its real-world application.

Wed 4 NovMachine Learning
The gist
Figuring out the best way to gather data from experiments is tricky, and researchers often build their own custom programs to do it. The authors created TestBED, a free toolkit that breaks down experiment design into easy parts anyone can mix and match. This helps teams use, compare, or improve methods more easily and tests new ideas. They also used TestBED to compare popular methods and found some surprising details about what works best. This toolkit aims to help more people use smart experiment planning in the real world.
Open → 2611.06633v1