Bayesian optimization runs faster and more accurate on cube regions
Acquisition for Spherical-Linear Bayesian Optimization on the Cube
Machine Learning
Summary
Optimizing complex systems often requires picking the best settings within certain boundaries, like a box shape (cube). The authors studied ways to speed up a popular method called Bayesian optimization when working inside a cube-shaped space. They found new ways to choose where to look next that are much faster while still being as accurate or better. Their method works well especially when tests take a moderate amount of time, making it a better choice in many practical situations.
What this means in practice
- •For machine learning engineers: Optimize hyperparameters of models faster with less computational overhead when the search space is a cube.
- •For simulation developers: Speed up automatic tuning of simulation parameters where each trial takes moderate computation time and inputs lie in cube domains.
Authors
Nathan Wycoff
Abstract
Recent work demonstrated that executing Bayesian Optimization (BO) using simply a linear surrogate with a clever transformation is enough to approximate the performance of the standard Gaussian-Process (GP) based version on high-dimensional test functions. When the decision region is a sphere, this strategy admits rapid exact acquisition. However, it is the unit cube we encounter most often in the BO literature, and in this setting a standard approach, namely a multi-start quasi-Newton method, is employed for solving the acquisition subproblem. In this article, we will discuss a series of results, both positive and negative, regarding the complexity of global resolution of various acquisition subproblems over the unit cube. On the basis of this discussion, we will propose a new acquisition function which is globally optimizable using significantly less execution time than the incumbent approach while maintaining or improving performance. The resulting method is competitive with GP-based BO in terms of accuracy per evaluation, and is superior in terms of accuracy per walltime on black-boxes of low to moderate computational intensity.