Stable method prunes neural network layers to keep accuracy
Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers
Machine Learning
Summary
Deep neural networks have many connected layers, and cutting down unnecessary parts can make them faster and smaller. The authors show a way to prune these layers one at a time instead of all at once, which keeps the network more stable and avoids cutting too much. Their method looks at small subnetworks inside each layer and carefully removes parts without hurting the network’s accuracy. This approach works well on different tests, including big language models.
What this means in practice
- •For machine learning engineers: Reduce the size of pretrained neural network layers more reliably to improve inference speed without large accuracy loss.
- •For ai infrastructure teams: Implement layerwise pruning to avoid unstable model reductions during deployment of large pretrained models.
Authors
Charles Kulick, Armenak Petrosyan, Sui Tang
Abstract
We propose a decoupled, layerwise method for structurally sparsifying the fully connected layers of pretrained neural networks. Rather than penalizing all layers jointly, our approach extracts shallow two-layer subnetworks, normalizes the inner weights, and applies a structured group penalty to the outer weight matrix of each block, processing layers sequentially to prune neurons and reduce the width of each layer. We prove that the constrained decoupled objective is equivalent at optimality to a specific joint penalty on the inner and outer weights, for any positively homogeneous activation, and thus admits a clean projected and proximal formulation. Our central finding is that this decoupled reformulation is more robust than coupled methods. In numerical experiments it provides a wider usable range of the regularization strength and a lower rate of catastrophic over-pruning than the tested joint baseline while maintaining comparable accuracy. We establish these properties in controlled classification and sparse-recovery studies, and examine their scope in a high-dimensional PINN stress test and in the feed-forward layers of OPT-1.3B.