VExpert improves load balancing for distributed AI models by virtualizing storage
vExpert: Virtualizing Expert Storage for Adaptive Load Balancing in Distributed MoE Inference
Distributed, Parallel, and Cluster Computing
Summary
Large AI models called Mixture-of-Experts (MoE) divide tasks among different special parts called experts. The problem is these experts are fixed in one place, but the work they get changes a lot, causing some experts to be overloaded while others sit idle. The authors created vExpert, a method that makes a flexible copy of these experts in memory and quickly switches between them to keep workloads balanced. This approach uses a model to predict delays and manage memory better, leading to faster overall performance in experiments.
What this means in practice
- •For cloud service operators: Improve efficiency and speed when running large AI models distributed across multiple GPUs by dynamically balancing workloads using virtual expert storage.
- •For high-performance computing teams: Reduce memory overhead and transfer delays in multi-node GPU clusters running AI inference by managing expert storage virtually and adjusting placement in real time.
Authors
Wenxun Wang, Xiuhong Li, Yida Wang, Chen Tang, Zongle Huang, Ke Hong, Yu Wang, Yongpan Liu
Abstract
Mixture-of-Experts (MoE) models have become prevailing, yet their distributed deployment suffers from severe dynamic load imbalance due to the conflict between static expert placement and inherent routing dynamism. While host DRAM offers capacity to avoid static device storage for load balancing, we identify existing offloading works fundamentally fail in distributed inference due to two overlooked challenges: (1) coupled performance impacts of runtime Host-to-Device transfers due to load dynamism and shared PCIe topology; (2) system-level memory overhead incurred by DRAM usage at scale. In this paper, we propose vExpert that shifts DRAM offloading paradigm from partial residency to full redundancy through virtualized expert storage. We pioneer the first systematic performance model that jointly captures H2D latency, PCIe contention, and load imbalance, building a lightweight allocator to dynamically map physical experts to virtual slots runtime. A novel disaggregated manager is designed to reduce memory overhead in multi-process environments. Evaluated on DeepSeek-V3.1 within an 8-node H100 cluster, vExpert improves load balance ratio by up to 55% and achieves 1.2$\times$ end-to-end speedup.