Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
2026-07-06 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors developed a new method called CAP to improve how large Mixture of Experts (MoE) models run across multiple GPUs and computers. CAP smartly places and removes parts of the model to reduce the amount of time spent communicating between devices, balancing speed and accuracy. It groups experts that work together often, adjusts how much work each device does, and prunes less important connections. Their tests show CAP speeds up model inference by about 1.2 to 1.9 times while keeping accuracy high compared to existing methods.
Mixture of Experts (MoE)Model inferenceGPU computingDistributed systemsExpert pruningCommunication overheadExpert placementThroughput optimization
Authors
Xiao Shi, Yingying Sun, Jiangsu Du, Zhiguang Chen, Yutong Lu
Abstract
As MoE models scale to hundreds of experts, placement and pruning decisions increasingly dictate communication volume, affecting the performance of distributed inference across GPUs and nodes. We propose CAP (Communication-Aware Assignment and Pruning), a framework that considers computation, communication and accuracy together for efficient MoE inference through expert placement and pruning. It consists of three components: (1) Co-activation driven expert placement, which groups frequently co-activated experts to reduce inter-device and inter-node communication; (2) Communicationcomputation trade-off adjustment, which generates placements with different computational load and communication volume; and (3) Communication-aware expert pruning, which selectively removes routing destinations to reduce communication with limited accuracy degradation. By combining these components, CAP selects an efficient operating strategy for different hardware configurations. Across our single-node and multi-node experiments, it achieves 1.23x - 1.86 x throughput improvement over DeepSeek EPLB and sequential placement in vLLM, and preserves better model accuracy at the same target speedup under lossy acceleration.