Geo distributed scheduling cuts data center carbon with little latency cost
Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Large language models like those that power chatbots use a lot of electricity, which causes carbon emissions. The authors studied how spreading the work across data centers in different places can save carbon by using cleaner energy at some sites. Because language models need fast responses, the authors made a detailed simulator that mimics delays caused by request queues and batching. They created and tested smarter ways to send requests to different data centers to save carbon while mostly keeping delays within limits. Their best method cuts up to 60% of carbon emissions with only a small increase in slow responses.
What this means in practice
- •For data center operators: Reduce carbon emissions of LLM inference by scheduling workloads based on regional grid intensity while respecting latency requirements.
- •For cloud infrastructure engineers: Implement batch-level schedulers that optimize workload distribution across sites with varying carbon footprints to meet strict latency targets.
Authors
Burak Topcu, Musa Cim, Zuhal Ozturk, Mahmut Taylan Kandemir
Abstract
Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions. Although carbon-aware scheduling can reduce these emissions in geo-distributed datacenters by exploiting the spatiotemporal variations in grid carbon intensity, the latency-sensitive nature of LLM serving constrains this design space. While prior studies investigate the carbon savings achievable through spatial scheduling, they oversimplify the dynamics of modern inference engines, such as queueing and continuous batching, which strongly shape inference latency. We address this issue by designing and evaluating a latency-centric, geo-distributed inference simulator, calibrated with hardware-grounded latency and power measurements, that reproduces these dynamics and enables the study of carbon savings under strict service-level objectives (SLOs). We then formulate a series of SLO- and carbon-aware schedulers of increasing sophistication, from per-request heuristics to batch-level optimization: (i) a greedy heuristic, (ii) an integer linear programming (ILP)-based scheduler jointly assigning batches of requests, (iii) an iterative ILP-based solution projecting its own assignments back into the load estimates to capture load coupling, and (iv) a facility mixed-integer linear programming (MILP) scheduler pricing the energy of batched requests collectively rather than in isolation. Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon-SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy, with an average SLO violation of 3%. Our characterization further shows that savings grow with the spread of carbon intensity across datacenters and shrink as system load increases, while SLO attainment depends on how frequently schedulers receive datacenters' load updates.