Network depth improves training speed for difficult learning tasks
Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
Machine Learning
Summary
Training deep learning models usually gets better with more time and data, but it's unclear if making the network deeper always helps speed up training. The authors show that for complex tasks, adding depth to a network can make training faster by improving how performance scales with time. For easier tasks, depth does not significantly change the training speed, only causing small log-scale effects. They also found a way to describe training progress that works for all task difficulties and depths, unifying the understanding of how networks learn.
What this means in practice
- •For machine learning engineers: Design network architectures that adjust depth based on task difficulty to improve training efficiency on complex tasks.
- •For ai model trainers: Use predictor norm based time reparameterization to better monitor and predict training progress across different network depths and task complexities.
Tested on one dataset.
Authors
Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
Abstract
Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.