Weight decay shapes neuron behavior in two layer neural networks

Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks

Machine LearningArtificial Intelligence

Summary

Weight decay is a common technique used to keep neural network parameters small, but how it guides neurons to align or 'condense' was not clearly understood. The authors study a simple two-layer neural network and find that training with weight decay happens in three stages: fast learning, shrinking of neuron strengths, and finally neurons aligning their directions. This alignment, called neuron condensation, emerges from the dynamics of a mathematical field that controls neuron behavior. The work shows weight decay does more than just shrink parameters—it actively encourages neurons to cluster in direction.

What this means in practice

  • For machine learning engineers: Optimize training regimes to encourage neuron alignment for improved network interpretability and efficiency using weight decay insights.
  • For ai model deployers: Better understand the role of weight decay in models to diagnose and control prediction stability during deployment.

A theory result. No direct application yet.

Authors

Cheng Xu, Pengxiao Lin, Zhangchen Zhou, Zhi-Qin John Xu

Abstract

Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.