Backbone contrastive pruning keeps neural networks accurate when sparse
BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks
Machine Learning
Summary
Making neural networks extremely sparse by removing many connections usually causes them to lose important information and perform poorly. The authors propose Backbone Contrastive Pruning (BaCP), which helps the network keep good internal representations by comparing them to versions of the network saved earlier or trained longer. This method improves accuracy when networks are very sparse and avoids big drops in performance. BaCP works well across many different pruning setups and criteria.
What this means in practice
- •For machine learning engineers: Maintain accuracy in heavily pruned models by preserving important internal features during extreme compression.
- •For edge device developers: Deploy compact neural networks that remain effective despite severe pruning for resource-limited hardware.
Authors
Mohammad Haroon Khawaja, Muhammad Haseeb, Mohammad Fatim Shoaib, Muhammad Tahir
Abstract
Unstructured pruning at extreme sparsity often suffers from representational collapse, causing sharp drops in accuracy. To address this, we study Backbone Contrastive Pruning (BaCP), which regularizes the sparse network's embedding space by aligning it with pretrained, fine-tuned, and historical snapshot models. Building on the contrastive decomposition of the CAP framework (Xu et al., 2022), we provide a rigorous matched-budget characterization of this approach across multiple pruning criteria. Evaluated across 90 settings, BaCP improves accuracy substantially in extreme sparsity regimes where standard pruning fails, and is close to baseline where representations remain intact.