Mode-switching network cuts large language model communication costs sharply
MoSE: Mode-Switching Expander for Mixed LLM Training and Inference
Networking and Internet Architecture
Summary
AI systems that run large language models often need to handle two very different kinds of work: training the models and using them to answer questions, called inference. Each type of work sends data differently inside the computer network, making a single fixed network design inefficient. The authors designed MoSE, a network that can switch its connections to better match the kind of work being done, reducing communication delays and costs. Tests showed that MoSE makes data transfers much more efficient during both training and inference without changing hardware.
What this means in practice
- •For data center operators: Improve communication efficiency when running mixed training and inference workloads on AI clusters by dynamically switching network connections.
- •For network architects: Design reconfigurable network topologies that adapt edge allocation to workload types without adding ports or changing routing.
Authors
Fan Yang, Ying Zhou, Binglei Wang, Zhenjie Zhou, Jialong Li
Abstract
AI clusters increasingly run large language model (LLM) inference and training on the same fabric. Prefill-decode (P-D) disaggregation creates key-value (KV) cache transfers between prefill and decode groups, whereas training collectives and all-to-all traffic benefit from near-uniform global connectivity. A static sparse topology can therefore be poorly matched to one of the two traffic patterns. We present Mode-Switching Expander (MoSE), a reconfigurable expander that treats topology design as a fixed-degree edge-allocation problem. MoSE reallocates the same sparse edge budget toward direct P-D connectivity in inference-heavy modes and restores a uniform random regular expander in training-heavy modes. We evaluate MoSE using a 1024-group flow-level topology model, shortest-path routing, and two mixed workloads. Across 20 seeds, MoSE reduces average and 95th-percentile (P95) load-aware KV communication cost by 90.8\% and 91.9\% relative to Static-Training in the inference-heavy mode. In the training-heavy mode, it reduces average and P95 training communication cost by 22.7\% and 27.6\% relative to stale Static-Inference. These results show that coarse-grained topology switching can support both workload modes without additional ports or routing changes.