Structured pruning improves neural networks using response geometry
Unified Response Geometry for Structured Pruning
Computer Vision and Pattern Recognition
Summary
Neural networks can be made smaller by removing parts called channels, but simply ranking these channels can miss important interactions. The authors propose a method that looks at groups of channels together to find those that keep the network’s performance best. They also provide a way to update the network without retraining after removing channels. Their method works better on a popular image recognition network, keeping more accuracy even after removing many channels.
What this means in practice
- •For machine learning engineers: Optimize deep learning models by selectively removing channels to maintain accuracy while reducing complexity without fine-tuning.
- •For embedded systems developers: Create efficient neural network implementations for resource-limited devices by pruning redundant parts with minimal accuracy loss.
Authors
Kaixiang Shu
Abstract
Structured pruning is commonly formulated as ranking individual channels, although channel responses can be complementary or cancel through downstream mixing. Motivated by these response interactions, we formulate pruning as the selection of a subset with large joint response capacity, followed by a separate functional realization step. Our unified response geometry maps each candidate set to \(M(D,R)=D^{1/2}RD^{1/2}\) and uses its determinant together with Schur-greedy residuals to select non-redundant coordinates. The same construction yields two information-conditioned instances: an unlabeled instance based on activation covariance, and a task-conditioned instance that combines activation and gradient variance for response scale with gradient correlation for complementarity. To convert the selected subset into an executable network, we fold predictable removed responses into successor weights through ridge compensation and recalibrate batch-normalization statistics, without fine-tuning the network. On ImageNet ResNet-50, the unlabeled instance reaches \(65.4\%\) and \(53.9\%\) Top-1 accuracy at 30\% and 40\% deletion, versus \(59.8\%\) and \(43.1\%\) for strength-only selection; the task-conditioned instance reaches \(67.7\%\) and \(56.3\%\) under the same protocol. A six-family screen shows architecture-dependent behavior, with positive relative contrasts in several convolutional and expansion-layer settings and clear boundary cases in windowed attention. These results support response geometry as a conditional principle for structured pruning, with its benefit determined jointly by the observed response and the architecture in which that response is realized.