Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how well residual neural networks can approximate different functions depending on their size and structure. They determined the exact minimum width needed in the network blocks to approximate functions in various mathematical senses, showing it depends mainly on the input and output dimensions. They also found limits on how small the network can be while still maintaining good approximation power. These results apply to common activation functions like ReLU and its variants, and highlight the importance of certain architectural choices in residual networks.
residual neural networksuniversal approximationblock widthLeakyReLUReLUuniform approximationLp approximationactivation functionsnetwork architecture
Authors
Qi Zhou, Xuan Zhou, Xiao-Song Yang
Abstract
In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions $d_x$ and $d_y$, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the block width are established. To achieve $L^p$ approximation $(1\leq p <+\infty)$ on any compact domain, we show that the exact minimum block width is $\max\{d_x,d_y\}$ when the inner width is 1. Furthermore, we show that residual neural networks with block width $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that residual neural networks with block width less than $\max\{d_x, d_y\}$ cannot approximate all target functions, both in the $L^p$ sense and the uniform sense, regardless of inner width.