Papers for

software architects designing neural network models

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ReLU networks with one layer are closed but two layers are not

On the (non-)closedness of the set of ReLU networks

Abstract: We prove that the set $\mathcal{N}_N$ of ReLU networks with one hidden layer and at most $N\in\mathbb N$ neurons is weakly closed in $W^{1,p}(Ω)$, for a bounded connected open set $Ω\subset\mathbb R^d$. Consequently, every coercive, weakly lower semicontinuous energy on this space attains its minimum over $\mathcal{N}_N$. This covers the Deep Ritz method with convex integrands, Sobolev training and Lipschitz learning. Moreover, as the weight decay parameter tends to zero, minimizers of the regularized problem converge, up to subsequences, to minimizers with minimal parameter norm. We also prove that the set of ReLU networks with two hidden layers of bounded widths is not closed in $W^{1,p}(Ω)$ or in $C(\overlineΩ)$. This disproves a conjecture of Petersen, Raslan and Voigtlaender.

Fri 6 NovMachine Learning

The gist

The paper finds that certain simple neural networks with one hidden layer behave nicely when viewed in a math space used for functions, meaning solutions to optimization problems exist within this set. However, neural networks with two hidden layers do not have this property, disproving a previous guess by other researchers. This result affects how we understand and work with neural networks when training with certain mathematical techniques. The authors also show that as certain penalties shrink, solutions approach those with the smallest weight norms.

Open → 2611.09887v1