Structured pruning reduces model size while keeping accuracy intact
Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning
Machine LearningArtificial Intelligence
Summary
Deep learning models can be too large to run efficiently on small devices, so people try to shrink them by removing parts that seem less important. This paper introduces a method called Artificial Structure Function Search (ASF-S) that picks which parts to remove based on how the network's structure affects its function, inspired by how brains work. The method uses a new way to measure importance called Principle Gradient Importance (PGI) that helps keep the model's output organized. The authors show that their approach can cut the number of parameters by 70% without losing accuracy and without needing to retrain the model.
What this means in practice
- •For mobile app developers: Deploy smaller deep learning models on phones while maintaining accuracy and reducing computation without extra retraining.
- •For embedded systems engineers: Create efficient neural network implementations that fit limited hardware by pruning layers while preserving functional output organization.
Authors
Mindula Illeperuma, Rafael Pina, Charuka Herath, Sharmarke A. Gabayre, Varuna De Silva
Abstract
Structured pruning is a model compression technique that is used to reduce the computational cost of deploying deep neural networks on resource-constrained devices. Popular methods of pruning rely on opaque heuristics or weight-based criteria that give no indication as to the structural dependencies in the network. To address these limitations we present Artificial Structure Function Search (ASF-S): a novel structured pruning framework. ASF-S utilizes Principle Gradient Importance (PGI): a novel prune-candidate selection criteria that is inspired by structure-function relationships in the brain. By ensuring the pruned structure of the model respects topographical organization of the output layer, we define Artificial Functional Connectivity (AFC) for artificial neural networks. AFC provides evidence to demonstrate that accurate smaller networks can be found using careful prune candidate selection criteria. We present results for PGI as a selection criterion and for ASF-S as a pruning framework against recent benchmarks, demonstrating that our method yields model variants with 70\% parameter reduction, that can recover baseline accuracy without re-training the pruned layers.