ReLU networks approximate smooth functions with new error limits

Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

Machine Learning

Summary

Approximating smooth functions accurately is important for many computer tasks. The authors studied how well neural networks made from simple building blocks called ReLUs can mimic these smooth functions while keeping their layer weights limited. They extended known results to measure errors not just in maximum difference but also considering how the function changes, using what's called Sobolev norms. Their findings show how network width, depth, and weight limits affect accuracy, especially for deep networks without restricting the function’s smoothness. This helps understand the trade-offs when designing neural nets for approximating complex mathematical functions.

What this means in practice

  • For machine learning engineers: Design neural networks with guaranteed error limits on smooth function approximation for tasks needing precise control of function derivatives.
  • For computational scientists: Use neural networks with constrained weights to approximate solutions to differential equations while quantifying error in function and derivative norms.

A theory result. No direct application yet.

Authors

Xianjun Li, Yunfei Yang

Abstract

Recent studies have shown that smooth functions can be well approximated by ReLU neural networks with path norm constraint on the weights. We extend these results from uniform approximation to approximation in Sobolev norm. Specifically, we analyze how well Sobolev functions in $W^{n,p}$ can be approximated by neural networks with width $W$, depth $L$ and path norm bounded by $K$, when the approximation error is measured in the $W^{1,p}$-norm. For shallow networks with depth $L=1$, we derive the approximation error bound $\mathcal{O}(\max\{W^{-(n-1)/d}, K^{-(n-1)/(s-n)}\})$, when the smoothness index satisfies $n<s=(d+3)/2$ and the input is $d$-dimensional. For deep networks, we remove the restriction on the smoothness by showing that the approximation bound $\mathcal{O}(K^{-(n-1)/(d+d/p+1)})$ holds if the width $W$ and depth $L$ are sufficiently large.