Summary
Neural networks that use the ReLU function usually have many layers to do their job. This paper shows that any ReLU network can be copied exactly using just two hidden layers. The authors give two different proofs, one using ideas from an area called homological algebra and another using direct counting methods. This means complex neural networks might be easier to understand or build.
What this means in practice
- •For neural network engineers: Design neural networks with only two hidden layers when exact reimplementation of any ReLU network is needed.
- •For machine learning system architects: Simplify neural network architectures without losing function by replacing deep ReLU networks with just two hidden layers.
A theory result. No direct application yet.
Authors
Florestan Brunck, Christoph Hertrich, Kilian Ruess, Jack Stade, Amir Yehudayoff
Abstract
We prove that every $\operatorname{ReLU}$ network can be exactly simulated by a $\operatorname{ReLU}$ network with two hidden layers. We present two proofs: one that is based on fillings in homological algebra and a direct combinatorial proof.