RankLane speeds up mixture-of-experts model prefill on AI hardware

RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

Machine Learning

Summary

Large language models sometimes use mixture-of-experts (MoE) to handle more data by activating only some parts called experts for each word. The authors found that existing methods waste time on padding or overflow because expert capacities are fixed and don’t adjust to current workload. RankLane rearranges how experts are assigned during processing to better match their load without changing device setup, reducing wasted work. This approach leads to faster model prefill times on specific AI accelerator chips compared to traditional methods.

What this means in practice

  • For ai hardware engineers: Speed up prefill phases of large MoE language models on fixed-capacity accelerator hardware by matching expert assignments to their dynamic workloads.
  • For cloud ai infrastructure teams: Reduce overall processing time and inefficiencies when running MoE models on accelerator clusters through improved routing and overflow recovery.

Authors

Wentao Ni, Chihao Yu, Hong Cai, Mingu Lee, Hejia Zhang, Fatih Porikli, Jishen Zhao

Abstract

Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.