Two hidden layers suffice to compute maximum of many inputs
Two hidden layer suffice: the caterpillar net
Machine Learning
Summary
Figuring out the maximum value from many inputs is important in understanding certain neural networks. The authors present a new design, called the Caterpillar Net, that calculates this maximum using only two layers of neurons. Their method uses a specific construction and has a clear geometric meaning. This confirms earlier work showing two layers are enough, but this approach is simpler and nonrecursive.
What this means in practice
- •For deep learning architects: Design neural networks with only two hidden layers that compute maximum functions efficiently for model simplification.
- •For optimization algorithm developers: Create optimization routines that use minimal layered neural structures for maximum value computations in piecewise linear contexts.
A theory result. No direct application yet.
Authors
Jakob Paul Zimmermann
Abstract
Every continuous piecewise linear (CPWL) function is a finite signed sum of maxima of affine functions, making maximum representation a central problem in the study of the expressiveness of ReLU networks. A now-disproved conjecture predicted logarithmic depth for maxima. Inspired by the proof by Brunck et al. (2026) that two hidden layers suffice for general CPWL functions, we give the Caterpillar Net, a nonrecursive construction for the maximum of $N$ inputs with two hidden layers of widths $O(N^2)$ and $2^{O(N\log N)}$. The construction has a direct geometric interpretation.