Relative Repairability: A Calibration-Based Diagnostic for High-Sparsity Post-Pruning Allocation
2026-05-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explore how pruning very sparse neural networks not only decides which connections stay but also where the damage caused by pruning happens and if that damage can be fixed easily. They introduce a measure called Relative Repairability (RR) that predicts how much damage remains after a simple repair step, using only unlabeled data. Their experiments show that RR is especially helpful when the network is at a critical point where usual pruning methods start failing but repair is still possible. This means that choosing which connections to prune should consider not just which weights remain but also how easily the damage can be repaired.
neural network pruningsparsityRelative Repairabilityactivation distortionchannelwise variance matchingrepair procedureResNet18CIFAR10CIFAR100magnitude based pruning
Authors
Qishi Zhan, Liang He, Minxuan Hu, Ziheng Chen
Abstract
At very high sparsity, neural network pruning does more than decide which weights remain. It also determines where pruning induced damage is placed across the network, and whether that damage can be recovered by a fixed lightweight repair procedure. We study this problem through the lens of repair conditioned sparsity allocation. We introduce Relative Repairability (RR), a calibration based diagnostic that compares the raw activation distortion caused by layerwise pruning with the residual distortion left after channelwise variance matching repair. RR estimates the fraction of local damage that remains after repair, using only unlabeled calibration data. Across ResNet18, ResNet34, and VGG16 BN on CIFAR10 and CIFAR100, we find that RR is not a universally dominant allocation rule. Instead, it is most useful near an architecture dependent recoverability transition, where standard structural or magnitude based allocation priors begin to lose reliability but post repair recovery has not yet fully collapsed. On CIFAR100 ResNet18, a fine grained sweep shows that RR improves over ERK across the central transition band and surpasses LAMP near the upper part of this band. A projection forced ablation further shows that capped ERK can over protect projection layers, shifting excessive sparsity onto regular convolutions and reducing post repair recovery. These results suggest that high sparsity pruning should allocate not only retained weights, but also repairable damage.