Routing queries across language models cuts energy use sharply
Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
PerformanceDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models use a lot of energy, especially when always running the largest model. The authors test a way to send questions to different sized models based on how hard the question is, saving energy without losing much accuracy. They train a system to pick the right model using previous data on speed, energy use, and correctness. Their experiments show this routing improves energy efficiency while keeping answers good.
What this means in practice
- •For cloud service operators: Reduce operational energy and cost by routing user queries among multiple models based on query difficulty without losing answer quality.
- •For mobile app developers: Improve app energy use by selecting smaller language models for simpler queries while reserving larger models for complex tasks.
Authors
Muhammad Abdur Rab Siddiqui, Daniela Rojas, Chen Yang, Wenqi Cui, Yuanyuan Shi, Yize Chen
Abstract
Large language models (LLMs) and agentic AI systems are creating rapidly growing inference energy demands as model sizes grow and reasoning trajectories extend. While in practice, many queries do not require the capabilities of the largest available model, and routinely directing such queries to a high-capability model can introduce unnecessary, considerable computation and energy consumption. In this paper, we investigate whether adaptive routing across a heterogeneous pool of LLMs can reduce this energy burden without substantially compromising task performance. We design a language-model-based router that reads in each query and selects an answer model from a fixed candidate pool. The candidate models are first profiled through an offline tournament that records their correctness, latency, power, and GPU energy for each query. Using these measurements, the router is trained through supervised fine-tuning followed by group relative policy optimization (GRPO) with the tailored paradigms. Results demonstrate that learned routing can selectively allocate expensive model capacity based on query context and improve the accuracy-energy tradeoff in multi-LLM serving. Across seven benchmark tasks, we also observe a sharp accuracy-energy phase transition among routers, providing practical insights into improving energy efficiency while maintaining LLM performance.