Purlin improves gpu communication for faster distributed inference
Purlin: Separating Orchestration from the Datapath of Collectives
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
When multiple GPUs work together on tasks like running AI models, they need to share information quickly and efficiently. The authors found that existing communication methods mix up what needs to be done, when and where, and how data moves, making it hard to adapt to new hardware. They created Purlin, a system that separates these parts, allowing better customization and use of advanced hardware features. Testing showed Purlin makes communication faster on various GPUs and improves performance for AI tasks like language models and image generation.
What this means in practice
- •For ai infrastructure teams: Increase efficiency of GPU communication in distributed model inference to speed up large language model serving and online AI tasks.
- •For gpu hardware designers: Customize collective communication methods to better leverage new hardware features and improve data movement for GPU workloads.
Authors
Osayamen Jonathan Aimuyo, Swapnil Gandhi, Christos Kozyrakis
Abstract
Distributed inference depends on GPU collective communication that must keep pace with evolving hardware and specialized workloads. However, existing collective implementations often couple semantics, orchestration (where and when data moves), and the datapath (how data moves). This coupling makes it costly to adopt new hardware mechanisms and customize communication for applications. We present Purlin, a scale-up communication framework that separates these concerns. At the top of Purlin, we specify collectives as a naming of an input and output layout and a copy or reduction operation. In the middle, we introduce a shared orchestration protocol, Stage, Notify, And Consume (SNAC), which derives coordination from these specifications. Below SNAC sits a hardware-specific datapath we call Atom, which implements two key data movement primitives for collectives: copy and reduce. This separation lets us customize collectives and adopt new hardware mechanisms while reusing orchestration via SNAC. We evaluate Purlin on A100, H200, and B200 GPUs. Across seven collectives, Purlin achieves latency speedups of up to 5.14x and bandwidth improvements of up to 4.50x over baselines. Integrated into SGLang, Purlin improves offline LLM serving throughput and interactivity by 1.13x on average and up to 1.37x over baselines. For online LLM inference, Purlin improves interactivity by 1.26x on average and up to 2.85x, with the largest gain occurring under overload. For diffusion image generation, Purlin reduces end-to-end latency by up to 1.13x.