Node-based flow control greatly speeds up data handling in high-demand networks
MailRail: Mitigating Incast Congestion with User-Space, Node-Based Flow Control
Distributed, Parallel, and Cluster Computing
Summary
When many computer processes try to send data at once, it can cause traffic jams that slow everything down. The authors show that controlling data flow by sharing limits across an entire computer node instead of controlling each process separately helps avoid these jams. Their system, called MailRail, lets many processes send data more smoothly and quickly, even when demand is very high. Tests show it can make data flow over a thousand times faster in tough situations while keeping normal workloads running efficiently.
What this means in practice
- •For cloud infrastructure teams: Manage data flow effectively in large server clusters to prevent slowdown caused by many simultaneous data requests.
- •For distributed machine learning engineers: Improve communication efficiency for high-demand AI models like GNNs and MoE by reducing congestion during parallel training.
Authors
Noel Gomillion, Lance Fletcher, Trevor Steil, Eun Jung Kim, Roger Pearce
Abstract
Irregular workloads, such as those stemming from Graph Neural Networks (GNNs) and Mixture-of-Experts (MoE) models, frequently induce severe incast congestion in large-scale parallel environments. Existing fabric-level congestion control mechanisms rely on reactive, link-level rate limiting that degrades overall throughput. While receiver-driven, credit-based flow control offers a proactive alternative, as process counts scale memory slots per rank fall below usable thresholds, rendering traditional rank-based accounting untenable. To address this challenge, we introduce MailRail, an out-of-order, asynchronous messaging framework, along with a novel user-space, node-based credit flow control system. By shifting credit accounting from individual processes to shared per-node pools, our proactive, fabric-agnostic approach maintains sufficient credit availability even as process density grows, enabling processes to dynamically utilize freed credits without waiting for receiver-side redistribution. Experimental evaluations demonstrate that node-based flow control achieves up to a 1132x speedup over unconstrained execution at an adversarial incast ratio of 4096:1 while adding minimal overhead on uniform workloads.