Gaussian process method controls variance to improve additive models
R2-GP: Variance-Calibrated Priors for Additive Gaussian Processes
Machine Learning
Summary
When using Gaussian processes to model data that combines multiple effects, standard settings can overestimate how much each part contributes to the overall pattern. The authors introduce a new way to set these contributions so that each part's expected effect on the model's accuracy is clear and balanced. Their method adjusts for how smooth or rough each part is and works well even with many parts or large data sets. They tested it and found it predicts better and recovers model parameters more accurately than previous methods.
What this means in practice
- •For statistical modellers: Build more accurate additive models by precisely controlling component variance in Gaussian process priors.
- •For machine learning engineers: Speed up Gaussian process inference on large data by using a calibrated prior combined with efficient kernel spectral approximations.
Authors
Javier Enrique Aguilar, Soham Mukherjee, David Kohns
Abstract
We develop R2-GP, an R2D2-type prior for Gaussian process (GP) generalised additive models (GAMs) that calibrates explained variance ($R^2$) for Matérn covariance kernels. The function variance for standard Matérn kernels depends jointly on the effect magnitude and length-scale. As such, smooth and rough components with identical amplitudes contribute very different amounts of variation. With independent priors on component amplitudes, this variation results in variance overestimation, especially in higher dimensions. We overcome this by taking direct control over each component's expected contribution to prior explained variance. We do this by rescaling each component based on its length scale and observed inputs, and modelling each rescaled variance as a share of the function variance induced by a global $R^2$ prior. This allows R2-GP to scale well to many components. To also scale to large sample sizes, we extend R2-GP to Hilbert space approximation methods, a framework for kernels exploiting their spectral decompositions for fast and efficient inference. Our simulations and real-world case studies demonstrate superior generalisation and parameter recovery over independent variance priors.