Bayesian optimization improves searching for functions with sparse kernels
Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds
Machine Learning
Summary
Optimizing unknown functions that take entire functions as inputs is hard because those inputs can be infinitely complex. The authors propose a new method that simplifies this problem by focusing on functions represented with only a few basic building blocks (kernels). This approach searches efficiently by adjusting where these basic blocks are placed and how strongly they influence the function. They tested their method extensively against previous techniques and showed that it works better across many examples.
What this means in practice
- •For machine learning engineers: Optimize machine learning models with functional inputs by efficiently searching function spaces for best parameter functions.
- •For roboticists: Tune control policies that are functions over time or space by optimizing in infinite dimensional spaces with sparse kernels.
Authors
Davide Sartor, Meghan E. Huber, Donghyun Kim, Nathan Wycoff
Abstract
Bayesian Optimization (BO) has become an established methodology for minimizing black-box functions of a vector input. Often, however, this parameter vector arises from the discretization of an inherently functional relationship. Several recent articles have considered the Functional Bayesian Optimization (FBO) setting, in which the variable to be optimized is not a member of a finite dimensional vector space, but rather an infinite dimensional function space. In this work, we propose $L^0$ Manifold Optimization (L0MO), a simple approach to FBO which searches the subset of a Reproducing Kernel Hilbert Space (RKHS) consisting of functions with a sparse representation in the kernel functions, optimizing both the kernel locations and their coefficients. We discuss in detail the relationship between our method and existing ones, providing a unifying lens through which to view prior works. To assess our method against the state of the art, we conduct an extensive computational study, and along the way develop a novel set of benchmark test functions which port standard finite-dimensional ones to the infinite dimensional domain. Our experiments demonstrate that, on balance, the proposed method achieves superior performance across a wide range of test benchmarks.