CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how to shrink big language models by cutting out whole parts like attention heads and feed-forward groups. They point out that previous methods look at each part separately, ignoring that some parts may be important only when considered together. Their new method, CoCurve, measures both individual importance and how pairs of parts interact using a mathematical tool called the Fisher matrix. This lets them decide what to prune all at once without retraining or extra data. The result is a more accurate way to reduce model size while considering interactions between parts.
Structured pruningLarge language modelsAttention headsFeed-forward network (FFN)Residual connectionsFisher matrixTaylor expansionToken-level KL divergenceModel compressionQuadratic programming
Authors
Zhiren Gong, Zihao Zeng, Zijie Wang, Tiantong Wang, Chau Yuen, Wei Yang Bryan Lim
Abstract
Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.