ReLU networks with one layer are closed but two layers are not
On the (non-)closedness of the set of ReLU networks
Machine Learning
Summary
The paper finds that certain simple neural networks with one hidden layer behave nicely when viewed in a math space used for functions, meaning solutions to optimization problems exist within this set. However, neural networks with two hidden layers do not have this property, disproving a previous guess by other researchers. This result affects how we understand and work with neural networks when training with certain mathematical techniques. The authors also show that as certain penalties shrink, solutions approach those with the smallest weight norms.
What this means in practice
- •For machine learning engineers: Ensure existence of optimized one-layer ReLU networks when minimizing energy with convex loss functions in certain function spaces.
- •For software architects designing neural network models: Avoid relying on theoretical closure properties for two-layer ReLU networks when constructing optimization or regularization frameworks.
A theory result. No direct application yet.
Authors
Eloi Martinet, Leon Bungert
Abstract
We prove that the set $\mathcal{N}_N$ of ReLU networks with one hidden layer and at most $N\in\mathbb N$ neurons is weakly closed in $W^{1,p}(Ω)$, for a bounded connected open set $Ω\subset\mathbb R^d$. Consequently, every coercive, weakly lower semicontinuous energy on this space attains its minimum over $\mathcal{N}_N$. This covers the Deep Ritz method with convex integrands, Sobolev training and Lipschitz learning. Moreover, as the weight decay parameter tends to zero, minimizers of the regularized problem converge, up to subsequences, to minimizers with minimal parameter norm. We also prove that the set of ReLU networks with two hidden layers of bounded widths is not closed in $W^{1,p}(Ω)$ or in $C(\overlineΩ)$. This disproves a conjecture of Petersen, Raslan and Voigtlaender.