Amortized approach improves thompson sampling for design simulators
Amortized Thompson Sampling and Efficient Information-Directed Sampling for Simulator-Based Design
Machine Learning
Summary
Choosing the best action when you're unsure can be tough, especially when you need to try many possibilities. The authors developed a method that prepares the hard work ahead of time so decisions can be made faster once new information comes in. This method also helps pick actions that teach us the most about what’s best, balancing learning and doing. They showed their approach works well in designing biological systems using complex simulators.
What this means in practice
- •For synthetic biology engineers: Speed up and improve experiment design by using efficient decision strategies with simulator models lacking tractable likelihoods.
- •For machine learning practitioners: Accelerate Bayesian optimization in continuous action spaces where traditional Thompson sampling is computationally expensive.
Authors
Michal Kobiela, Michael U. Gutmann
Abstract
Thompson sampling is a widely used strategy for sequential decision-making under uncertainty, offering a principled way to balance exploration and exploitation. This strategy involves maximizing a target objective under a single sample drawn from the current belief over unknown quantities. Conventionally, this optimization is performed once the belief has been inferred from data, and it becomes computationally burdensome when many Thompson samples are required, especially in continuous action spaces. In this work, we introduce an approach that amortizes this process ahead of time, before any data is collected, so that an approximation of the full Thompson sampling distribution is available immediately once observations arrive. We employ this approach to solve the multi-round setting, involving sequential data acquisition and Thompson-sample recommendation across rounds. Beyond accelerating Thompson sampling, our approach enables efficient estimation of expected information gain about the optimal action. This can be used to select more informative Thompson samples or to obtain maximally informative designs with optimization. Combining this estimate of expected information gain with immediate regret supports information-directed sampling (IDS), which can outperform Thompson sampling by explicitly balancing these two quantities. We demonstrate our approach on two simulator-based design problems in synthetic biology, where the simulators have intractable likelihoods.