Bayesian optimization gets much faster using linear models
Fast Bayesian Optimization Through Linear Models
Machine Learning
Summary
Bayesian optimization is a tool used to find the best settings or answers when testing each option is costly or slow. The authors noticed that existing methods are usually slow but accurate, while some fast methods are less effective. They developed new faster Bayesian optimization techniques that work well even with many variables, running 100 to 1000 times faster without losing much accuracy. This advance lets people use Bayesian optimization in situations where speed really matters.
What this means in practice
- •For machine learning engineers: Speed up hyperparameter tuning of large-scale models by using these fast Bayesian optimization methods with minimal accuracy loss.
- •For control system designers: Optimize parameters in real-time control systems faster while maintaining performance using the proposed linear-model-based Bayesian optimization.
Authors
Colin Doumont, Henry Moss, Philipp Hennig
Abstract
Bayesian optimization (BO) is typically used only for problems where the underlying function is deemed "expensive" (e.g. in time or money), with the speed of the BO algorithm being of little importance. In the real world, however, problems span a continuous range of values, going from cheap to expensive. In this paper, we present BO algorithms through the lens of a speed-performance Pareto frontier: we compare BO against faster algorithms like CMA-ES, and highlight how certain regions of the Pareto frontier are currently empty. As a solution, we propose a new set of fast BO methods that exploit the linear structure in existing BO algorithms, for both continuous and combinatorial inputs (from ~10 to 6000+ dimensions). Our methods achieve performance similar to state-of-the-art BO, while being more than $10^2\times$ to $10^3\times$ faster, opening the field of BO to an entirely new class of problems.