Structured pruning method improves large language model efficiency
Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) work well but need lots of memory and computing power, making them hard to run efficiently. The authors identify a problem where the part of the model that tries to recover lost functions after pruning is too simple to fully regain what was removed. They created a method called OverRep that trains a bigger, more complex recovery part temporarily, then shrinks it back down without losing performance. This approach keeps the model small and fast when used, but better at keeping its smarts after pruning.
What this means in practice
- •For machine learning engineers: Deploy smaller LLMs that maintain better reasoning ability with reduced memory and compute costs using this recovery method after pruning.
- •For cloud infrastructure teams: Run pruned LLM models more cost-effectively at scale without sacrificing accuracy on reasoning tasks by using compact inference modules.
Authors
Seungmin Oh, Donggeon Lee, Jongbin Ryu
Abstract
Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.