High dimensional smoothness enables stable function extrapolation
Into the danger zone: stable extrapolation in high-dimensional function and operator learning
Machine Learning
Summary
Predicting outcomes outside the data you've seen before is usually hard and unreliable. This paper shows that for certain smooth mathematical functions and operators, it's possible to make stable and reliable predictions even when testing on very different data from training data. The reason is that functions become smoother in higher dimensions, which actually helps with prediction accuracy. The authors prove this mathematically and confirm it with computer simulations using methods like deep neural networks and polynomials.
What this means in practice
- •For machine learning engineers: Improve robustness of ML models when deployed on data differing significantly from training data by exploiting smoothness properties in high-dimensional inputs.
- •For computational physicists: Predict outcomes of physical systems beyond observed regimes by applying stable extrapolation techniques grounded in high-dimensional smoothness assumptions.
Authors
Ben Adcock, Simone Brugiapaglia, Xuemeng Wang
Abstract
Out-of-distribution (OOD) generalization is a central challenge in scientific machine learning. We study regression problems in which the test distribution differs from the training distribution and ask: under what assumptions on the target function or operator is stable extrapolation possible, and how far beyond the training domain can one extrapolate? Existing theory controls the test error through additive penalties measuring the discrepancy between the training and test distributions. Such guarantees show robustness to small distribution shifts, but can very pessimistic in comparison to OOD performance observed empirically. We identify classes of holomorphic functions and operators for which the OOD generalization error converges at algebraic rates even in the presence of large distribution shifts. This phenomenon stems from the increasing smoothness of higher-index coordinates, leading to what we term a `blessing of high dimensionality'. For learning with either polynomials, deep neural networks or deep neural operators, we derive explicit rates for arbitrary test measures supported on suitable domains and quantify how the admissible domain depends on the underlying regularity of the function or operator. Our extrapolation guarantees are independent of the test distribution, depending only on its support. We also present a series of numerical experiments across a range of functions and operators that support the main theoretical findings.