Efficiently select experts to speed up large AI models
EAT: Expert Account Tracker for Efficient MoE Inference
Artificial Intelligence
Summary
Large AI models called Mixture-of-Experts (MoE) use many smaller modules called experts to make decisions, but activating too many experts wastes time and resources. The authors introduced EAT, a method that keeps track of how well each expert has performed in the past to smartly choose only the most useful ones for each task. This reduces the number of experts that need to be activated by over 25%, making the model faster without losing accuracy. They also show that the model’s performance can be recovered quickly if experts are pruned too much.
What this means in practice
- •For machine learning engineers: Improve the speed and efficiency of large-scale Transformer models by dynamically activating fewer expert modules.
- •For cloud ai infrastructure teams: Optimize operational costs by reducing computation during MoE model inference through history-aware expert selection.
Authors
Yuexian Li, Yifei Yang, Zouying Cao, Hai Zhao
Abstract
Mixture-of-Experts (MoE) models have emerged as a revolutionary method to scale Transformer models. However, traditional MoE architecture still suffers from inefficiency since a large number of experts are unnecessarily activated. Existing approaches for reducing the number of activated experts often overlook the historical performance of each expert. In this paper, we propose EAT, a novel method called Expert Account Tracker (EAT), which utilizes history-awareness metrics and adaptive thresholding to dynamically select the most important experts, thereby reducing the activated expert number while effectively maintaining the model performance. Experiments show that EAT outperforms the existing baseline Top-P method across multiple models and datasets, achieving over 25% an average reduction compared to the vanilla method in the number of activated experts and performing better token generation speed compared to the baseline. Furthermore, the performance of pruned models can be efficiently recovered via OPD using only 9K data. Additionally, through ablation studies, we find that excessively reducing the number of activated experts can significantly harm model performance, and the importance of experts varies across layers, with higher-level experts being generally more critical.