Distributed load balancing reduces AI service delays by up to 17 percent
DLB: Distributed Load Balancing at Scale for Generative AI Inference
Distributed, Parallel, and Cluster Computing
Summary
Generative AI services need powerful but expensive hardware like GPUs to run quickly. Usual methods to share these computers’ work can be slow or inefficient, causing delays. The authors designed DLB, a smart system that spreads AI tasks across many computers in different locations while keeping track of how busy each one is. This helps deliver responses faster, reducing delays by up to 17% on average, even under heavy and varied workloads. The system has been in use at Google for nearly two years, handling millions of AI requests every second.
What this means in practice
- •For cloud infrastructure teams: Manage AI workload distribution efficiently across data centers to reduce user wait times and infrastructure overprovisioning.
- •For network schedulers: Use real-time server capacity data to route diverse AI inference tasks with lower latency at scale.
Authors
Santiago R. Balseiro, Bartek Wydrowski, Sameer Agarwal, David Applegate, Aaron Archer, Soheil Hassas Yeganeh, Alex Iriza, Bobby Kleinberg, Balasubramanian Sivan, Pranav Vaish, Oscar Zegarra, Wenxin Zhang, Vahab Mirrokni, Amin Vahdat
Abstract
The reliance on scarce and expensive accelerators such as GPUs and TPUs in modern datacenters places unprecedented demands on backend infrastructure. For workloads characterized by heterogeneous service times and complex multi-stage processing, such as Generative AI, conventional load balancing techniques are often inadequate, relying heavily on costly overprovisioning to maintain service level objectives. This paper introduces DLB, the Distributed Load Balancer, a novel system designed to minimize end-to-end user latency for large-scale, heterogeneous workloads. DLB employs a scalable, distributed design with peer-to-peer probing to maintain real-time visibility into server capacity across large-scale, geographically distributed infrastructure. The system continuously learns latency models to estimate the latency impact of routing decisions, allowing it to effectively manage heterogeneous hardware and diverse model architectures. We provide a novel theoretical analysis of our routing algorithms that establishes their stability and global performance guarantees over time. We also evaluate DLB through extensive simulations, which show substantial gains compared to state-of-the-art load balancing algorithms. Finally, following a 22-month deployment of DLB at Google, where it facilitates large-scale Generative AI inference for thousands of different machine learning models and millions of requests per second, we detail the design choices and practical experiences gained from the system in production. Analysis of production migrations demonstrates that DLB yields statistically significant latency reductions compared to the legacy baseline, including a 17\% decrease in median latency and a 13\% decrease at the p95 tail.