MoE inference runs faster by overlapping requests without guessing experts
Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models that use many specialized parts, called experts, can be run on local computers by loading experts as needed. The authors found that trying to guess which experts will be needed soon and preloading them can waste memory and bandwidth. Instead, they built a system named Bevy that runs some requests ahead while waiting for others to load missing experts, effectively overlapping work without needing to guess. This approach increases throughput and makes better use of the memory cache on GPUs.
What this means in practice
- •For machine learning engineers: Run large MoE language models locally with higher inference speed by using Bevy's scheduling and expert fetching method.
- •For cloud infrastructure operators: Improve throughput of MoE model serving systems by interleaving concurrent requests to reduce GPU stalls without relying on expert prediction.
Authors
Hongseung Yu, In Gim, Kyunghan Lee, Lin Zhong
Abstract
Expert offloading enables large mixture-of-experts (MoE) language models to run locally, but conventional batching stalls the GPU while missing experts are fetched. A popular remedy, known as prefetching, is to predict what experts will be needed soon and fetch them using idle link bandwidth. Imprecise prediction, however, wastes bandwidth and GPU memory on fetching unnecessary experts. Our key finding is that prefetching can be harmful when there is an expert cache in GPU memory. This is because as the cache includes more necessary experts, expert prediction becomes less precise. Our key insight is that proper request scheduling beats prefetching. Building on this insight, we present Bevy, an MoE inference system based on runahead inference for concurrent requests. A request whose experts are available can run ahead to later layers while other requests wait for fetches, revealing the experts it needs next. Bevy can then fetch the missing experts so identified while computing ready requests, overlapping computation and fetching without predicting expert selections. Running requests separately, however, repeats reads of shared weights in GPU memory and increases compute time. To limit this cost, Bevy's self-balancing scheduler batches ready requests at each layer and prioritizes lagging requests, adapting their grouping as expert availability changes. Across two local platforms, Bevy delivers up to 3.46$\times$ the decode throughput of llama.cpp and up to 1.55$\times$ that of the fastest prefetching baseline.