ReLU networks approximate smooth functions with optimal error rates

Optimal approximation of smooth functions by ReLU neural networks

Machine Learning

Summary

Approximating smooth functions accurately with neural networks is important but often inefficient. The authors show how to construct ReLU neural networks that approximate these smooth functions as well as theoretically possible, improving on previous approaches. Their method uses cleverly designed network architectures and mathematical techniques to capture local details of the functions. This leads to better predictions from data in statistical tasks without extra penalties.

What this means in practice

  • For machine learning engineers: Design neural network models for smooth data that achieve the best possible uniform approximation error, enhancing model accuracy efficiently.
  • For statistical modelers: Use specially constructed clipped ReLU networks to reach optimal rates when estimating smooth function relationships from observed data.

Authors

Thomas Nagler

Abstract

We study uniform approximation of the Hölder ball $\mathcal H_1^s([0,1]^D)$ by fully connected ReLU networks. Except in the one-dimensional Lipschitz case, existing constructions remain separated from the information-theoretic efficiency limit. This article closes the gap for every $s>0$ and $D\ge1$: networks with width $O(N)$, depth $O(L)$, and parameter magnitude $O(1)$ attain uniform approximation error of order $(N^2L^2\log N)^{-s/D}$. No relation between width and depth is required. The construction uses new encoding and decoding mechanisms that exploit higher-order finite-difference bounds for local Taylor coefficients of the target function. As a statistical consequence, empirical risk minimizers over the corresponding clipped network classes attain the minimax regression rate $n^{-2s/(2s+D)}$ without additional logarithmic factors. The paper also gives a detailed exposition of the key approximation techniques, covering both established and new mechanisms.