Bayesian optimization improved with information geometry and trust regions
Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods
Machine LearningArtificial IntelligenceInformation Theory
Summary
Bayesian optimization helps find the best options when you can't try everything. The authors study this by looking at the geometry of the information involved, which helps understand why some methods struggle with many variables. They use this insight to create a new way to adjust how the optimization explores options by measuring sensitivity locally. Their method, called FITR, works well in tests and can adapt to different kinds of problems.
What this means in practice
- •For machine learning engineers: Design better optimization routines for tuning models with many parameters by using local sensitivity measures instead of fixed scaling.
- •For autonomous systems developers: Improve adaptive decision-making in robotics or control systems by integrating trust-region Bayesian optimization that handles complex sensitivity patterns.
Authors
Saksham Kiroriwal, Julius Pfrommer, Jürgen Beyerer
Abstract
We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.