Geometry reveals limits of fixed-size ReLU neural networks
The Geometry of Limits in ReLU Function Spaces
Machine Learning
Summary
This paper looks at what kind of shapes and functions can come from certain neural networks with a fixed design using ReLU activations. The authors found rules about how these functions behave when they get close to limits, showing that some expected smoothing properties actually fail in certain networks. They also pinpoint specific ways networks break down when trying to approximate some functions, giving a new way to understand the boundaries of what these neural nets can represent.
What this means in practice
- •For neural network designers: Avoid architectures that fail to approximate certain function limits by understanding where ReLU networks' function spaces are not closed.
- •For machine learning tool developers: Improve network training diagnostics by detecting blow-up residues indicating degenerate parameter configurations in ReLU networks.
A theory result. No direct application yet.
Authors
Kathryn A. Lindsey
Abstract
We study the geometry and closure of spaces of functions realized by ReLU neural networks of fixed architecture. We first establish a general result about limits of continuous piecewise-linear (CPWL) functions: the minimum number of convex sets on which a function is affine is lower semicontinuous under pointwise convergence. The proof uses a connection to hypergraph coloring and a compactness principle, and yields quantitative bounds on the polyhedral complexity of continuous limits. We then characterize the functions realizable by shallow ReLU networks with vector-valued outputs, extending a known scalar-valued characterization. Using this and the geometry of limiting hyperplane arrangements, we prove that fixed-width shallow scalar ReLU realization spaces are closed in the weighted supremum norm on the whole input space. In contrast, we construct an explicit sequence of networks with architecture $(2,3,2\mid1)$ whose limit is not realizable by that architecture. Non-closedness holds both globally in weighted supremum norm and under uniform convergence on compact sets with nonempty interior, disproving a conjecture of Petersen, Raslan, and Voigtlaender. Finally, we show that non-closedness for a fixed scalar-output ReLU architecture necessarily involves a \emph{blow-up residue}, in which diverging output coefficients amplify an emerging linear dependence among hidden features. Together, these results provide geometric and functional-analytic tools for studying the boundaries and degenerations of ReLU realization spaces.