Layer wise pruning method improves large language model efficiency at high sparsity
Building a Nested LADDER for LLM Unstructured Post-Training Pruning
Machine Learning
Summary
Pruning large language models means removing some parts to make them smaller and faster, but this can hurt how well they understand and generate text. The authors created LADDER, a method that prunes models carefully by testing small changes in each layer to see which cause the least damage. This step-by-step pruning keeps the model working better, especially when a lot of pruning is done. They tested LADDER on several popular language models and found it keeps accuracy higher than other methods when removing many parameters.
What this means in practice
- •For machine learning engineers: Optimize language models for deployment by pruning more parameters with less accuracy loss at high sparsity levels
- •For cloud platform operators: Manage model compression more effectively to reduce compute costs while maintaining service quality for large language models
Authors
Nicolás Mauricio Cuadrado Ávila, Ivo Gollini Navarrete, Martin Takáč, Samuel Horváth
Abstract
Non-uniform allocation can improve post-training pruning of large language models, yet quality can still degrade sharply at high sparsity. To control degradation as sparsity increases, we introduce LADDER (Layer-Wise Allocation via Damage-Driven Evaluation and Refinement), which allocates sparsity incrementally. At each step, it evaluates pruning an additional fraction of weights from each eligible layer and commits the layer update that causes the least end-to-end degradation per removed parameter. This process produces non-uniform sparsity profiles and exactly nested masks across multiple budgets. We evaluate LADDER on five OPT, Llama, and Qwen models across 10-90% sparsity, using SparseGPT as its candidate-generation backend and comparing against uniform and adaptive allocation baselines. Results reveal a clear operating-regime trade-off: LADDER is comparable or slightly weaker at benign sparsity but becomes substantially more robust from 70% onward. This high-sparsity advantage largely persists after posterior distillation across four of five models and translates into improved retention of aggregate zero-shot accuracy.