Improving input selection accuracy in gaussian process regression
Calibrated Derivative-Process Sensitivity for Gaussian-Process Variable Selection
Machine Learning
Summary
Choosing the right inputs is important when using Gaussian process models to make predictions, but the usual way of ranking inputs can be misleading. The authors propose a new calibrated method based on how sensitive the model’s output is to changes in inputs. They fix problems with standard tests by introducing a bootstrap technique, which helps to select important inputs more reliably while reducing false positives. This approach works well in simulations and real data, even with highly correlated inputs.
What this means in practice
- •For machine learning engineers: Select relevant features more reliably in Gaussian process models to improve prediction accuracy and interpretability in regression tasks.
- •For biostatistics teams: Control false discoveries when identifying influential covariates in medical prediction models using Gaussian processes, ensuring better inference.
Authors
Jia Cai
Abstract
Automatic relevance determination (ARD), the default tool for variable selection in Gaussian-process (GP) regression, ranks inputs by inverse lengthscales -- which measure how fast a function varies, not how much an input contributes to prediction -- and offers no calibrated rule for deciding which inputs to keep. The prediction-centred alternative, the derivative sensitivity $ν_j = \mathbb{E}[(\partial f/\partial x_j)^2]$, is available in closed form from a fitted GP, but turning it into a selection rule is harder than it looks: at a null input the estimator is a degenerate quadratic form, so Wald and Bernstein-von Mises cutoffs are anti-conservative, and the natural residual bootstrap is mis-scaled. We show that a studentized multiplier bootstrap of the GP derivative process repairs both, prove its validity through an invariance principle for quadratic forms, and obtain asymptotic family-wise and false-discovery-rate control across inputs. Over 100 replications the rule controls FDR wherever inputs are truly null, while uncalibrated derivative rankings breach the target by up to 2x and a Bernstein-von Mises cutoff by 2.2x; at matched FDR it loses no power; it holds under a Matérn kernel and input correlation up to 0.99; on real data with planted and authentic null inputs it admits 5-12x fewer spurious inputs; it costs 5-18% of the GP fit; and a block-averaged variant retains validity at cost linear in $n$.