Information based choices improve sparse gaussian process regression
On Basis Function Selection for Sparse Gaussian Process Regression
Machine Learning
Summary
Gaussian process models help predict data but can be slow with lots of information. To speed them up, people use a smaller set of building blocks called basis functions. Usually, they pick the first few basis functions without checking if they matter. The authors suggest smarter ways to pick only the important basis functions using ideas from information theory, which can make predictions better and more efficient. They tested these ways on common datasets and found improvements, especially for certain types of basis functions.
What this means in practice
- •For machine learning engineers: Improve prediction speed and accuracy in Gaussian process models by selecting only relevant basis functions based on information criteria.
- •For data science teams: Enhance model training on large regression datasets using smarter basis function choices to reduce computational cost without losing accuracy.
Authors
Marnix Van Soom, Ivan De Boi
Abstract
Sparse Gaussian processes achieve $O(N)$ inference by replacing the kernel with an appropriate expansion in a fixed basis $\{φ_j\}$ on the input space. Given a compute budget $M \ll N$, practitioners conventionally truncate the basis to its first $M$ entries. Nothing in the formalism, however, prevents one from selecting only those $M$ basis functions that matter for the data at hand. This would avoid spending budget on basis functions where there is no signal, but it requires a criterion for ranking the candidates. We propose three such criteria derived from an information-theoretic view of the basis-function selection problem. Each criterion matches a different state of knowledge at selection time: a no-data state, a no-prior state, and an in-between state. We then study the performance of truncation versus selection strategies on six UCI regression benchmarks across three basis families: Hilbert-space Gaussian processes (HSGP), variational Fourier features (VFF), and variational inducing spherical harmonics (VISH). We observe that the no-data criterion is a safe default, matching or improving on truncation for HSGP, VFF and VISH, with substantial gains for VISH and improvements over a recently developed selection heuristic for that basis family. The data-aware no-prior and in-between criteria provide substantial gains over truncation specifically for HSGP, which is the most broadly used of the three families in practice.