Gradient efficient gaussian process prediction with reduced cost

Derivative Gaussian Processes on a Two-Direction Budget

Machine Learning

Summary

Gaussian processes can make predictions about unknown functions and using gradient information can improve their accuracy. However, using full gradients is often expensive in terms of computation and memory. The authors propose a new method that uses only two directional derivatives per gradient, greatly reducing this cost while maintaining accuracy. Their approach allows more scalable and efficient predictions with gradient data, even beating some methods that only use function values. This means it can handle bigger problems with less computer power.

What this means in practice

  • For machine learning engineers: Create scalable surrogate models that incorporate gradient info more efficiently to speed up training and prediction of complex functions.
  • For computational scientists: Run accurate Gaussian process-based emulations of expensive simulations with gradient data while reducing memory and compute resources.

Authors

Hyunseok Seung, Matthias Katzfuss

Abstract

Gradient observations promise more accurate Gaussian process (GP) surrogates, but the cost of incorporating them has long stood in the way of realizing that promise. We propose a derivative GP with a budget of just two directions per observed gradient. One direction focuses on each gradient's direct contribution to target prediction, while the other aggregates its indirect contributions through correlations with the conditioning function values. Within a Vecchia approximation, where each prediction conditions on $m$ nearby inputs in $d$ dimensions, this construction represents their $md$ gradient coordinates using at most $2m$ directional derivatives, giving $\mathcal{O}(m^3)$ dense factorization cost per prediction target. For general conditioning sets, we bound the posterior approximation error relative to using full gradients and characterize when the error is small or the approximation is exact. In simulations, our method matches the accuracy of a leading exact gradient-reduction method at equal conditioning set size. Because its cost grows much more slowly with that size, it can use conditioning sets well beyond the memory limit of the exact method, reaching lower prediction error with a small fraction of the time and memory. Notably, our method can exploit gradient observations while requiring less computation time or memory than function-only GP baselines.