Deep linear networks reveal how early learning improves predictions

Investigating the Early Signatures of Feature Learning in Deep Linear Networks

Machine Learning

Summary

Understanding how deep neural networks learn important features from data is often tricky. This paper studies a simplified kind of network—deep linear networks—right after the first step of training. The authors find exact expressions describing how these networks start learning useful features, especially when the data has clear signals hidden in noise. They show that the first training step can improve the model’s predictions only when the data’s important patterns are strong and clearly aligned with the task. This helps explain when and why deep learning begins to work well early in training.

What this means in practice

  • For machine learning engineers: Design better training schedules by understanding when early gradient steps improve feature extraction for high-dimensional data.
  • For data scientists in finance: Optimize predictive models on financial signals by identifying when low-dimensional structure in noisy data enhances learning from initial steps.

Tested on simulated data.

Authors

Shaunak Bhandarkar, Boris Hanin

Abstract

Deep neural networks learn data-dependent features, a capacity widely credited for their empirical success. However, explicitly characterizing what features emerge -- and when they improve generalization -- is challenging, since training is nonlinear and non-convex and the data is often anisotropic. Here, we give a closed-form account of feature learning for a deep linear network in a mean-field parameterization, trained by one gradient step from initialization on high-dimensional data carrying a low-dimensional (spiked) signal. In the infinite-width mean-field limit, one step updates each layer's feature kernel by a target-aligned rank-one spike, amplified by both the learning rate and the layer depth. We next assess how these features impact prediction through kernel ridge regression (KRR), finding that KRR on the learned feature kernel yields scalar inflation of the KRR estimator at initialization -- an "anti-shrinkage" mechanism that reinflates task-relevant directions suppressed by ridge, while also amplifying noise. To study when this inflation improves generalization, in the overparameterized proportional limit we derive closed-form expressions for the bias, variance, and generalization errors of these estimators in terms of key macroscopic parameters -- notably the spike strength and spike-target alignment. Crucially, we find that a one-step generalization advantage vanishes for isotropic inputs and appears when both the spike strength and an effective signal-to-noise ratio are sufficiently large, demonstrating that a successful first gradient step both requires and benefits from low-dimensional task structure.