Minimal neural network widths linked to geometric shapes in math
The minimal width of neural networks and immersions of real projective space
Machine Learning
Summary
This work looks at how narrow neural networks can be while still being able to approximate any continuous function from multiple inputs to multiple outputs. The authors connect this question to a concept in advanced mathematics called smooth immersions of real projective spaces. They prove that under certain mathematical conditions, the minimal width required depends on the dimensions of the inputs, outputs, and these special geometric shapes. This finding extends previous results and hints at a deeper link between neural network design and mathematical topology.
What this means in practice
- •For neural network architects: Identify theoretical lower bounds on network width to guide efficient model design for given input and output sizes.
- •For applied mathematicians: Use the linkage between topology and network capacities to explore new mathematical tools for analyzing deep learning architectures.
A theory result. No direct application yet.
Authors
BaoZhi Chu, Ben Weinkove
Abstract
We investigate the minimal width of deep neural networks that can approximate all continuous functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ in the topology of uniform convergence on compact subsets. We show that whenever the real projective space $\mathbb{RP}^{n-1}$ admits a smooth immersion into $\mathbb{R}^n$, the minimal width is at least $n+\min\{n,m\}$, assuming that the activation function is continuous and monotone. As a consequence, the minimal width of ReLU networks is $n+m$ whenever $n$ is equal to 2, 3, 4, 7 or 8 and $1\le m \le n$, extending results of Hanin-Sellke and Hwang. This suggests an intriguing connection between the approximation theory of neural networks and differential topology.