Neural networks achieve exact minimum size for precise function fitting
Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
Machine LearningNeural and Evolutionary Computing
Summary
This paper explores how to build the smallest possible neural networks that can accurately approximate a wide variety of smooth functions. The authors show that for functions with several inputs, the absolute minimum number of neurons needed is exactly the number of input variables plus one. They also propose specific ways to build such networks using special activation functions and coding techniques. Their results prove the theoretical limits of how compact and efficient neural networks can be for precise function approximation.
What this means in practice
- •For machine learning engineers: Design neural networks with guaranteed minimum neuron count for precise function approximation in high-dimensional tasks.
- •For embedded system developers: Optimize neural network size and encoding bit complexity to fit precise models on low-memory hardware platforms.
A theory result. No direct application yet.
Authors
Zilan Cheng, Li-Lian Wang, Zhongjian Wang
Abstract
We study the minimum number of hidden neurons required for arbitrary-accuracy approximation of multivariate Hölder-continuous functions on $[0,1]^d$ and the associated encoding complexity. For $d\geq 2$, we construct a fixed, explicitly defined activation function for which a closed-form network with two hidden layers of widths $d$ and $1$ achieves arbitrary accuracy in the uniform norm. We prove that $d+1$ is the exact minimum total number of hidden neurons among standard feedforward networks with locally integrable activations and affine outputs. We further give a simpler construction using a single elementary activation that combines the floor and exponential functions. This construction requires three hidden layers of widths $d$, $1$, and $2$, only two neurons above the minimum. If a skip connection is allowed, widths $d$, $1$, and $1$ suffice. These constructions use explicit grid addressing and integer encoding of quantized function values. For a bounded $α$-Hölder class, they require $O(\varepsilon^{-d/α}\log(1/\varepsilon))$ bits, matching the metric-entropy lower bound up to a logarithmic factor.