Razor prunes redundant experts in large language models to save storage
RAZOR: Pruning Replaceable Experts in LLMs
Machine LearningComputation and Language
Summary
Large language models often use many specialized components called experts, but keeping all of them requires a lot of storage. The authors developed RAZOR, a new way to identify which experts can be removed without hurting the model's overall output. RAZOR looks at how well remaining experts can replace those removed, without needing to retrain the model. Tests show RAZOR keeps the model’s performance better than other methods while removing up to half the experts.
What this means in practice
- •For machine learning engineers: Reduce storage and memory costs of deploying large language models by pruning experts while maintaining output fidelity.
- •For cloud platform operators: Lower infrastructure costs by running pruned mixture-of-experts models that need fewer resources without retraining.
Authors
Mingyang Song, Mao Zheng
Abstract
Mixture-of-experts (MoE) models activate few experts per token but store the full expert pool. Expert pruning reduces this storage burden; at a fixed pruning budget, the goal is to preserve the original model's output distribution as closely as possible. Yet an expert's usage or contribution magnitude does not by itself determine the damage caused by its removal. What matters is whether the surviving computation can replace its function. We introduce RAZOR, a training-free expert pruning method that scores functional replaceability using consensus residuals: deviations of expert outputs from the original weighted mixture. An exact single-deletion identity at a fixed layer input accounts for survivor renormalization and router-selected refill, providing local scores aggregated over calibration tokens for budgeted pruning without gradients or recovery training. On GLM-4.7-Flash, Qwen3.6-35B-A3B, DeepSeek-V4-Flash-0731, and Hy3 at 25\% and 50\% expert removal, RAZOR achieves the highest nine-task macro average among the evaluated pruning methods in all eight settings. On the two backbones with matched REAP benchmark runs, it exceeds REAP by 2.12--5.59 points and wins all 36 paired task comparisons. It also lowers reverse KL relative to REAP in all four matched GLM-4.7-Flash and Qwen3.6-35B-A3B model--budget settings. Analysis of responses generated by Qwen3.6-35B-A3B nevertheless reveals changes in diversity, formatting, and termination, underscoring that task retention and predictive fidelity do not ensure generation stability.