Affine geometry and kink patterns in neural networks change with training
Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
Machine Learning
Summary
Neural networks with ReLU activations divide their input into simple affine regions separated by 'kinks' where the output bends sharply. This paper studies how many such kinks and internal activation switches exist in randomly initialized networks and how these change after training. The authors develop formulas to exactly count expected switches and kinks, showing that training redistributes these features rather than just reducing them. This helps explain how the network's piecewise-linear structure adapts during learning.
What this means in practice
- •For machine learning engineers: Estimate how complexity in neural network features evolves during training to guide architecture and training decisions.
- •For medical data analysts: Use activation switch counts predicted by the formulas to analyze segment class separability in medical datasets.
Tested on one dataset.
Authors
Recep Özkan, Christian Hirsch
Abstract
We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.