Open source toolkit speeds up intelligent experiment design research
TestBED: An Open-Source Toolkit for Bayesian Experimental Design
Machine Learning
Summary
Figuring out the best way to gather data from experiments is tricky, and researchers often build their own custom programs to do it. The authors created TestBED, a free toolkit that breaks down experiment design into easy parts anyone can mix and match. This helps teams use, compare, or improve methods more easily and tests new ideas. They also used TestBED to compare popular methods and found some surprising details about what works best. This toolkit aims to help more people use smart experiment planning in the real world.
What this means in practice
- •For machine learning engineers: Develop and evaluate adaptive data collection strategies using a reusable modular platform that supports diverse methods and benchmarks.
- •For industrial r&d teams: Optimize experimental setups for product development by quickly comparing and testing different Bayesian design strategies within one toolkit.
Authors
Gavin Kerrigan, Marcel Hedman, Sahel Iqbal, Tom Rainforth
Abstract
Bayesian experimental design (BED) is a principled framework for intelligent data acquisition with applications across a wide range of scientific and industrial domains. However, the development and adoption of BED remains hindered by a lack of standardised software and benchmarks. Existing methods are typically implemented in bespoke codebases and evaluated on a small subset of tasks, making methods difficult to apply, compare, or extend. To address this, we present TestBED, a general-purpose, open-source toolkit for Bayesian experimental design. TestBED decomposes the BED process into modular, reusable components spanning tasks, objectives, training mechanisms, policy architectures, and evaluation procedures, together with scalable implementations of core primitives such as expected information gain estimators. We provide a wide range of these components, which can be freely composed, allowing existing methods to be reproduced within a common framework, while also enabling previously unexplored configurations and new methodologies to be easily implemented and evaluated. Using TestBED, we further conduct a large-scale benchmark of contemporary BED methods, including controlled studies of under-appreciated but surprisingly critical methodological choices, such as the importance of time embeddings in design policy architectures. By providing a modular, extensible software foundation and a reproducible benchmarking suite, we hope that TestBED will accelerate methodological research in BED and lower the barrier to its real-world application.