Sliding networks keep deep learning stable and diverse features
Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity
Machine Learning
Summary
Deep learning models work better when they are bigger and deeper, but making them bigger often forces a choice: either learn good features or keep a variety of different features. The authors propose a way to scale deep networks that keeps both good feature learning and diverse features at the same time. They test their approach with theory and experiments on smaller networks and language models, showing it balances learning well but can affect performance differently when reshaped. Their work helps understand how network size and shape affect learning quality and diversity.
What this means in practice
- •For deep learning engineers: Design deeper and wider neural networks that preserve stable training and good feature variety using the sliding network scaling approach.
- •For natural language processing teams: Adjust language model architectures to balance depth and width while maintaining effective feature learning and diversity.
Authors
Datong Zhou, Blake Bordelon
Abstract
Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal \emph{feature learning} and preserving a form of \emph{feature diversity} in the residual stream arising from random differences between channels. We examine a \textit{joint scaling recipe} that preserves both. With residual width $N_\flat$, hidden width $N_\sharp$, and depth $L$, the key quantity is, by default, the limiting ratio $N_\flat/(N_\sharp L)\toα$, while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive $α$ to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.