Carbon aware routing reduces emissions of language model calls
Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems
Artificial Intelligence
Summary
Using large AI language models for answering questions usually sends data to powerful cloud computers, which use a lot of energy and release carbon emissions. The authors created a system that decides whether to handle these requests on nearby smaller edge devices or on cloud servers based on the specific carbon impact and the task's difficulty. Their method predicts how well the answer will be, how fast it will be, and how much power it will use on different devices to choose the best one with the lowest carbon footprint. Tests show the system matches cloud accuracy while cutting carbon emissions by about four times.
What this means in practice
- •For cloud infrastructure teams: Route AI model queries dynamically between cloud and edge to drastically lower carbon emissions without losing accuracy.
- •For distributed systems engineers: Incorporate carbon-aware routing to optimize where AI inference requests run based on environmental and performance trade-offs.
Authors
Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas, Iraklis Anagnostopoulos
Abstract
Large Language Models (LLMs) with function-calling capabilities are becoming critical for modern agentic AI systems. Nevertheless, current deployments typically route inferences to powerful cloud-based models, incurring significant energy use and carbon emissions. We address this sustainability challenge with a carbon-aware routing framework that distributes function-calling queries across a three-tier edge-cloud architecture, combining edge and cloud LLMs on heterogeneous hardware. At its core, a lightweight k-NN predictor operating in a unified semantic-lexical embedding space estimates query-specific accuracy, delay, and power consumption on each edge tier. These predictions are then combined with real-time grid carbon intensity to route every query to the lowest-emission tier capable of executing it successfully. Evaluated on state-of-the-art function-calling benchmarks and LLM families, our framework matches cloud-level accuracy while reducing operational carbon emissions by $4\times$ on average.