Large language models cut costs by adapting layer expert usage
Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion
Machine Learning
Summary
Large language models are expensive to run because they do a lot of calculations. To save on these costs, the authors study a way to split parts of these models into smaller groups called experts and choose only some to use for each word. Instead of using the same number of experts for every part of the model, they propose adjusting this number differently for each layer, like spreading colors in a rainbow. This approach improves performance and reduces errors in multiple language tasks compared to using a fixed number of experts.
What this means in practice
- •For ai infrastructure engineers: Optimize large language model deployment by dynamically allocating expert usage per layer to reduce calculation costs and improve inference accuracy.
- •For machine learning ops teams: Implement adaptive expert selection to improve performance on downstream language tasks while managing memory and compute constraints.
Authors
Zachary Broveak, Christian Kubicka, Thierry Tambe
Abstract
Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.