Summary
Large language model web services use lots of electricity that creates carbon emissions, which vary depending on where and when the service runs. The authors found that moving conversations between locations can backfire because each part of the conversation depends on earlier parts saved in one place, making it expensive to switch. They designed CarbonRouter, which decides where and how to handle each message in a conversation by considering these costs to save carbon. Their tests showed CarbonRouter cuts emissions by over 16% compared to other methods while keeping answers good and responses fast, especially for longer conversations.
What this means in practice
- •For cloud service operators: Reduce carbon emissions by routing conversation queries to optimal regions, GPU types, and model sizes while minimizing costly conversation history recomputations.
- •For energy management teams: Integrate cache aware carbon cost models to better forecast and schedule workloads in geo distributed data centers for greener operations.
Authors
Xiaoyang Zhang, Yuejun Yan, Qingang Zhang, Yonggang Wen
Abstract
The sustainability of the Web has drawn growing attention, as large language model (LLM)-based web services leave a substantial carbon footprint. These services typically run in geo-distributed data centers whose grid carbon intensity varies across regions and over time, so routing each query to a cleaner region or a smaller model can reduce emissions. A conversation, however, is a stateful web session: each turn reuses the key--value (KV) cache of its history, which resides only where earlier turns were served. Serving a turn on another deployment requires recomputing this history, whose carbon cost grows with the conversation and can outweigh the saving of the move. Existing carbon-aware LLM routers treat a conversation's queries as independent and ignore this cost. We present CarbonRouter, an online framework that jointly selects the model, GPU type, and region for each query to minimize carbon emissions. CarbonRouter introduces a cache-aware carbon cost model, which charges history recomputation to every candidate deployment using cached-token counts from the serving backends, and uses a lightweight gate that keeps a session on its cached deployment when it would likely move back at the next turn and rebuild its history again. We evaluate CarbonRouter in trace-driven experiments on two real-world conversation datasets with five LLMs, three GPU types, and four regions, where it achieves the lowest carbon of all policies. CarbonRouter reduces CO$_2$ by 16.1\% on average relative to FairCarbon, a state-of-the-art carbon-aware router with the same action space, while also improving predicted answer quality and estimated latency. Moreover, the saving rises further to 23.1\% for sessions longer than 30 turns.