FastAFD speeds up mixture-of-experts decoding on modern NVLink racks

FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?

Distributed, Parallel, and Cluster ComputingMachine Learning

Summary

Decoding large language models efficiently is important for fast responses. The authors study a technique called Attention-FFN disaggregation (AFD), which separates parts of the model across different GPUs to improve speed. They build and test FastAFD on a particular modern GPU cluster, showing it can be up to 17% faster than traditional methods on some models. They also develop a simple model to predict when this method will be beneficial, helping decide if switching is worth it before trying. This approach is especially helpful in setups that allow overlapping operations or have lighter expert models.

What this means in practice

  • For large-scale ai engineers: Increase throughput of serving large mixture-of-expert language models on NVLink GPU clusters by using FastAFD to better overlap computations across GPUs.
  • For cloud service operators: Optimize deployment strategies for AI inference on rack-scale multi-GPU setups by predicting when disaggregation techniques like FastAFD yield speed gains.

Authors

Yichao Fu, Yuxuan Zhang, Minshen Zhang, Ruitian Wang, Junda Chen, Hao Zhang

Abstract

Attention-FFN disaggregation (AFD) separates attention from the experts of a mixture-of-experts (MoE) model and has been reported to raise decode throughput over colocated serving. Whether this advantage holds on rack-scale NVLink systems such as GB200 NVL72 remains unclear: prior results were obtained on other hardware, against narrower colocated baselines, or with analytical models, whereas production engines serve large MoE models on these systems with wide expert parallelism (wide EP). We build and tune FastAFD, an AFD engine for GB200 NVL72 that overlaps the MoE of one microbatch with the attention of the other, and compare it against the best vLLM wide EP setting. Experiments show that AFD has a throughput advantage of up to 1.17x over wide EP on Qwen3 and 1.16x on Qwen3.5, and matches it on MiniMax-M2.5. We derive an analytical model that predicts AFD throughput within 3% using only profiles of colocated vLLM, telling whether switching to AFD would pay off before deploying it. The model yields a concise insight: AFD's speedup factors into a batch gain (larger batches, since attention GPUs hold no expert weights), an overlap gain (shorter steps, since the MoE runs on the expert GPUs behind attention), and a disaggregation cost (the GPUs given to experts). AFD has the advantage when the two gains outweigh the cost. This view explains which models benefit from AFD, and shows that speculative decoding helps light-expert models most, where the enlarged overlap gain outweighs the larger disaggregation cost and adds a further 8% to AFD's advantage over wide EP. Looking ahead, serving the experts on accelerators that keep their weights in on-chip SRAM, such as the Groq LPX accelerators paired with Vera Rubin NVL72, would remove most of the disaggregation cost, raising the ideal speedup to 2.48x with speculative decoding. FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.