MoE-Core enables efficient MoE model use in limited memory devices
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
Distributed, Parallel, and Cluster Computing
Summary
MoE (mixture of experts) models split tasks across many specialized parts called experts, which can reduce computing needs but use more memory. The authors propose MoE-CORE, a system that smartly manages these experts' memory use and data transfers to fit within small device memory. It carefully decides which experts to keep on the device and when to move experts in and out to speed up inference. Their approach drastically lowers the time to generate each token compared to prior methods, making it easier to run large MoE models on devices with limited memory.
What this means in practice
- •For ai systems engineers: Run large MoE model inference efficiently on constrained-memory AI devices by optimizing expert memory use and transfers.
- •For embedded device developers: Implement advanced language model features on small hardware by coordinating expert residency and data movement to fit limited memory.$Commercial implications: Enables compact AI appliances to offer practical MoE inference, supporting product offerings with improved scalability and speed.
Authors
Ke Yang, Yongji Gao, Xushi Li, Kui Luo, Sicheng Zhang, Tianming Zhou, Keyi Liu, Shufang Lu, Aoxuan Chen, Jie Meng, Jingchun Gao, Dan Li, Xinkai You, Dan Li, Zhixiang Xia, Yan Shi, Yang Liu, Yanjia Zeng, Liangjun Feng
Abstract
Sparse expert activation reduces MoE models' computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memory-constrained MoE inference. It stages complete expert layers in alternating buffers during prefill. During decode, it combines nonuniform layer-wise cache capacity, domain-informed initialization, routing-history-aware replacement, and cross-layer prefetching. The main configuration executes router-selected experts exactly; an optional score-based substitution path handles eligible low-score misses. The main comparison uses 1K- and 128-token output caps for MoE-CORE and vLLM Prefetch, respectively. Across five workloads per model, MoE-CORE records a mean time per output token (TPOT) of 38.0-44.8 ms versus 1268.9-1269.1 ms for the evaluated vLLM Prefetch configuration on DeepSeek-V4-Flash-W4A8; the corresponding values on GLM-5.2-W4A8C8 are 206.6-220.5 and 5941.5-5941.8 ms. Under an 84-GB NPU-memory cap, the best measured DeepSeek GSM8K configuration achieves a TPOT of 21.5 ms with approximate expert substitution and multi-token prediction (MTP) at depth 2. These results support coordinated expert residency and transfer scheduling under a device-memory constraint. The code is here.