Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy
2026-06-15 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to design special activation functions for neural networks that can closely approximate complex functions in mathematical spaces called Sobolev spaces, which measure both a function and its derivatives. They first show that a fixed-size neural network with their Elementary Universal Activation Function (EUAF) can approximate functions with second derivatives accurately. Then, they create a smoother activation function (DUAF∞) to extend this to higher-order function spaces. They also develop sigmoidal versions that maintain this approximation ability. Their work includes explicit formulas for how big and deep the networks need to be, using simple activation functions.
Activation functionsSobolev spacesNeural networksFunction approximationW^{s,∞}-normFixed-size networksElementary Universal Activation Function (EUAF)Differentiable Universal Activation Function (DUAF)Sigmoidal activationApproximation theory
Authors
Baicheng Li, Haizhao Yang, Shijun Zhang
Abstract
In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in $W^{2,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy, measured in the $W^{1,\infty}$-norm, by a fixed-size neural network using the Elementary Universal Activation Function ($\mathrm{EUAF}$). To extend this result to $W^{s,\infty}((a,b)^d)$ for $s\in\mathbb{N}$, we introduce a smooth activation $\mathrm{DUAF}_{\infty}$ from the family of Differentiable Universal Activation Functions ($\mathrm{DUAF}_n$). We prove that any function in $W^{s,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy in the $W^{s-1,\infty}$-norm by a fixed-size $\mathrm{DUAF}_{\infty}$-activated network. We further construct sigmoidal variants $\widetilde{\mathrm{DUAF}}_n$ and show that, for every $1\leq s\leq n$, fixed-size $\widetilde{\mathrm{DUAF}}_n$-activated networks still approximate any $f\in W^{s,\infty}((a,b)^d)$ with arbitrary accuracy in the $W^{s-1,\infty}$-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.