Structured pruning method improves large language model compression without calibration
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Machine LearningComputation and Language
Summary
Large language models are often too big and slow to use efficiently, so experts try to shrink them by removing less important parts. Existing ways to do this need extra data or training, which can be complicated and time-consuming. The authors of this paper introduce LILA, a new method that measures how important parts of the model are by comparing mathematical patterns in its weights, without any extra data or training. LILA prunes models effectively while keeping their original design and shows better or similar performance compared to other methods.
What this means in practice
- •For machine learning engineers: Reduce large language model size efficiently without extra calibration steps, simplifying deployment on hardware with limited capacity.
- •For natural language processing teams: Improve inference speed and resource use of language models by pruning unnecessary neurons based on spectral geometry without extra data requirements.
Authors
Sankar Behera, Dhruv Singh, Anshika Agnihotri, Raj Kumar Choudhary, Satyadev Ahlawat, Yamuna Prasad
Abstract
Structured pruning of large language models (LLMs) offers hardware-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (\emph{Latent-Informed Layer Analysis}) scores neuron importance via the Kolmogorov--Smirnov (KS) distance between empirical singular value distributions of the full and neuron-ablated feed-forward network (FFN) weight matrix, providing a closed-form spectral rule requiring no training, calibration data, or auxiliary network. Without any fine-tuning, LILA surpasses PruneNet (45M-parameter RL policy) by 1.57~pp in zero-shot accuracy on LLaMA-2-7B at 25\% sparsity, and outperforms WikiText-2-calibrated SliceGPT by up to 6.0~pp across all sparsity levels, while preserving the original architecture. After one epoch of LoRA recovery fine-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0.48~pp margin across LLaMA-2-7B and Phi-2, despite using zero calibration data. A Neural Tangent Kernel analysis confirms a 22$\times$ reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion. Finally, extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.