L0-regularized mixture-of-experts speeds up large language models
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts
Artificial IntelligenceComputation and Language
Summary
Large language models are very powerful but can be slow and costly to run. The authors designed a technique called L0-MoE that mixes different smaller expert models efficiently within a big model, speeding it up by up to 2.5 times without losing much accuracy. They used smart ways to organize data and train the model quickly. This approach performs better than other methods that try to speed up language models.
What this means in practice
- •For machine learning engineers: Accelerate inference of existing large language models in production systems while maintaining performance near original levels.
- •For cloud service providers: Reduce computational costs when serving large language models by integrating lightweight Mixture-of-Experts techniques into model deployment pipelines.
Authors
Zhenyu Zhang, Jiudong Yang, Zhaowen Tao, Meng Chen
Abstract
Large language models (LLMs) achieve strong performance but suffer from slow and costly inference. Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources. In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss. Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training. Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.