Activation sparsity training improves efficiency of large language models
SGA-Sparse: Training Native SwiGLU Gates for Activation-Sparse and Low-Bit LLMs
Machine LearningArtificial IntelligenceComputation and Language
Summary
Large language models use a lot of computing power, especially in parts called feed-forward networks (FFNs). This paper shows a way to train models so that only a fraction of these FFN parts need to be active during use, saving resources. The authors train a model called SwiGLU to decide early which parts to activate, making the model both sparse and able to use low-bit weights to reduce memory use. Their approach keeps the models accurate while reducing how much data needs to be processed.
What this means in practice
- •For machine learning engineers: Build resource-efficient large language model deployments by training models to activate only necessary neural channels and use low-bit weight representations.
- •For cloud infrastructure managers: Reduce hardware load and energy consumption in serving language models by applying activation sparsity and low-bit quantization together.
Authors
Ruokai Yin, Steven Li
Abstract
Activation sparsity can reduce FFN weight traffic during single-user LLM decoding, but active channels must be identified early enough to avoid fetching unused weights. In pretrained SwiGLU models, training-free routing with the native gate performs substantially worse than routing with the up activation at high sparsity. We show that this ordering reverses after sparsity-aware training. Based on this insight, we introduce SGA-Sparse, which converts pretrained SwiGLU LLMs into fixed-budget activation-sparse models by training the existing gate projection directly as a top-k router. SGA-Sparse updates all model parameters end-to-end while preserving the original SwiGLU activation and introducing no auxiliary router parameters. After approximately 30B training tokens, LLaMA models from 1B to 8B achieve uniform per-token 75-85% FFN channel sparsity while remaining within 0.2-2.7 pp of their original dense checkpoints in average zero-shot accuracy. Because post-training quantization degrades sparsity-trained checkpoints, we further study two complementary quantization-aware training (QAT) recipes: continued-QAT largely recovers performance at 4-bit weight quantization, while joint-QAT performs better for the more aggressive sub-4-bit target. To compare activation sparsity with dense low-bit quantization, we translate the sparsity-induced reduction in weight traffic into an equivalent bit-width. Under this unified weight-traffic metric, our 2.5- and 1.5-bit-equivalent models establish a new accuracy-traffic Pareto frontier: the 2.5-bit-equivalent models match dense 3-bit QAT accuracy, while the 1.5-bit-equivalent models approach dense 2-bit models and outperform dense ternary models by 1.3 pp on average. These results establish native-gate training as a practical approach for combining high FFN sparsity with low-bit weights.