Neural networks approximate smooth functions from infinite data spaces

ReLU Neural Network Approximation to Smooth Functional Operator: Dimensional Decay and Error Analysis

Machine Learning

Summary

This paper looks at how well deep neural networks with ReLU activation can approximate smooth functions that take infinitely many inputs organized in a certain mathematical space. The authors analyze which inputs are most important and use that to build efficient network approximations with provable error bounds. They find that when the input importance and sensitivity decrease exponentially, the neural network's approximation error also decreases rapidly, nearly optimally. This is the first time anyone has clearly connected the decay of input importance to approximation error for infinite-dimensional inputs.

What this means in practice

  • For machine learning engineers: Design neural networks that approximate smooth functional operators with guaranteed error bounds based on input coordinate importance.
  • For signal processing teams: Create models that handle functional data inputs with decaying coordinate influence using neural networks efficiently.

A theory result. No direct application yet.

Authors

Shuhao Jiao

Abstract

We study the uniform approximation of smooth scalar-valued functionals on an infinite-dimensional separable Hilbert space by deep ReLU neural networks. Writing the functional input as $X(t)=\sum_{d\geq1}ξ_dν_d(t)$, we quantify the importance of coordinate $d$ through $w_ds_d$, where $s_d$ bounds the magnitude of the corresponding basis score and $w_d$ controls the directional Fréchet sensitivity of the target functional. Our constructive analysis combines coordinate truncation, anisotropic partitioning, local Taylor approximation, and ReLU network realization, while allowing unrestricted interactions among the retained coordinates. We establish a general nonasymptotic upper bound for the uniform approximation error and a complementary pseudo-dimension-based lower bound for the worst-case approximation error. Under generalized exponential coordinate decay $w_ds_d\asymp\exp(-cd^ρ)$, with $ρ>0$, the upper and lower bounds match at the leading order and thus yield the nearly optimal approximation rate, which is stretched-exponential in the logarithm of the network budget. This is the first work to characterize neural network approximation error for infinite-dimensional functional inputs explicitly through the joint dimensional decay of coordinate magnitudes and directional sensitivities.
View PDFOpen arXiv