Bayesian optimization better tunes itself by optimizing its parameters
Pre-training of Bayesian Optimization Algorithm through Bayesian Optimization
Machine Learning
Summary
Bayesian optimization is a popular way to find good solutions to difficult problems but it needs some settings to be chosen beforehand. The authors suggest a way to improve these settings by simulating likely problem scenarios and testing how well different settings perform. They then use another Bayesian optimization process to find the best settings based on these tests. This method helps pick parameters that lead to better overall performance when solving problems.
What this means in practice
- •For machine learning engineers: Tune Bayesian optimization parameters in automated model selection for faster, more reliable results.
- •For algorithm developers: Use the framework to pre-train optimization algorithms to reduce manual parameter tuning effort.
Authors
Satoshi Katayama, Shoyo Hunt, Shintaro Masuda, Masayuki Karasuyama
Abstract
Bayesian optimization (BO) is widely used as a standard approach for expensive black-box optimization. However, BO algorithms often involve parameters that must be specified in advance, and their performance can strongly depend on these choices. We propose a framework for optimizing such parameters using sample paths drawn from a Gaussian process (GP) inferred from the information available at the start of BO. We use cumulative regret as the performance metric for a BO algorithm. By running the BO algorithm on the generated sample paths, we obtain an empirical estimate of its expected cumulative regret for a given parameter configuration. Optimizing this estimate allows us to identify parameter configurations that, given the currently available information, are expected to achieve low cumulative regret. Since this parameter optimization is itself a black-box optimization problem, we employ another BO procedure to solve it, which we refer to as outer BO. Through experiments, we demonstrate that the proposed framework can effectively select parameter configurations that achieve strong performance among a range of candidate configurations.