Hardware-aware pruning improves large language models on edge devices

Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models

Machine LearningHardware ArchitectureComputation and Language

Summary

Large language models can be too big and slow to run on small devices like phones or edge computers. The authors propose a new way to shrink these models by carefully removing parts of them while thinking about how the device stores and uses the remaining parts. Instead of applying the same shrinking pattern everywhere, their method picks the best way to prune each part of the model to save space and speed up responses without much loss in quality. They tested their method on real devices and popular models, showing it works better than simpler pruning methods.

What this means in practice

  • For edge device engineers: Run large language models more efficiently on resource-limited hardware by selecting pruning structures tuned to latency and memory constraints.
  • For cloud infrastructure teams: Reduce inference cost of language models by applying hardware-aware pruning that balances sparsity and performance across model sublayers.

Authors

Cameron Barker, Andreas Paraskeva, Jan N. van Rijn, Elliot J. Crowley, Henry Gouk

Abstract

Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically consider only the total number of retained weights. Consequently, their pruning decisions do not account for how target hardware stores and processes those weights. These methods select a sparsity structure in advance and apply it uniformly to all projections, rather than accounting for how each structure trades pruning error against latency and memory across sublayers. We propose Heterogeneous Structured Pruning (HSP), a hardware-aware pruning framework that combines two components: (i) a heterogeneous search space of dense, unstructured, 2:4, block-sparse, and head / channel (H/C) pruning options and (ii) structure-regularising constraints based on latency and memory models. Our contribution generalises existing importance-based methods: a global solver selects one option per sublayer to minimise proxy pruning loss under a resource budget, thereby assigning each projection a structure and density. We evaluate HSP on OPT and LLaMA models for an NVIDIA Jetson Orin Nano Super edge device, where it improves the quality-resource frontiers over training-free baselines. A project page and updated links for the related code is maintained at: cameronbarker.me/publications/hsp.