Neural network training outside teacher signals harms performance
Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation
Machine Learning
Summary
Training neural networks often involves learning meaningful patterns called teacher signals. This paper finds that parts of the network that learn unrelated patterns can create noise and worsen the network's accuracy. The researchers developed a method to isolate and understand this harmful effect during training. They also found that keeping these unrelated parts fixed can actually improve performance. This work helps explain why focusing learning on useful features is important for better results.
What this means in practice
- •For machine learning engineers: Improve neural network training by freezing some parts of first-layer weights to reduce noise and boost prediction accuracy.
- •For data scientists in signal processing: Better understand how unrelated feature learning inflates variance and degrades model generalization in shallow networks.
Authors
Samet Demir, Zafer Dogan
Abstract
We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically tractable surrogate whose asymptotic test-error dynamics coincide with those of SGD. In simulations with a ReLU student, freezing the directions of the orthogonal complement improves generalization relative to training them. Finally, we show that the signal component rapidly aligns with the second-layer weights, revealing strong cross-layer coupling even in this shallow setting. Together, our results provide a dynamical explanation for how feature learning outside the teacher subspace harms generalization.