Structured pruning helps large language models run faster and use less memory
Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning
Computation and LanguageArtificial Intelligence
Summary
Large language models work well but are expensive to run because they need a lot of memory and processing power. The authors found that a step called recovery after removing parts of the model can be tricky because the simple recovery parts might not understand everything that was removed. They created a method called OverRep that trains a bigger recovery part temporarily to learn better, then shrinks it back down for fast running without extra cost. This method improved how well the smaller models think while keeping resource use low.
large language modelsstructured pruningmodel recoveryoverparameterizationreparameterizationinference costreasoning performanceannealed activationmodel compressionTFLOPs
Authors
Seungmin Oh, Donggeon Lee, Jongbin Ryu
Abstract
Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.