Weight redundancy cuts transformer layers to speed up large models
Forward-Free LLM Depth Pruning via Weight Redundancy
Machine LearningArtificial IntelligencePerformance
Summary
Large language models use many layers that make them slow to run. Some methods remove whole layers to make the models faster, but they need trial runs on data to decide which layers to drop. The researchers created a way to find similar layers just by looking at the model’s stored weights, without running it on any data. This lets them pick layers to remove quickly and save computing time, working nearly as well as data-based methods.
What this means in practice
- •For machine learning engineers: Reduce running costs of large language models by pruning layers without extra calibration data or forward passes.
- •For cloud service providers: Cut inference latency and resource use for language models by automatically identifying redundant layers via weight analysis.
Authors
Vincent-Daniel Yun, Woosang Lim
Abstract
Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.