Edge servers balance load and cut wait times for AI language requests
End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
Artificial Intelligence
Summary
Large language models used in AI services need quick responses, so they are spread across many edge servers. But these servers have different speeds and can get busy in unpredictable ways, making it hard to pick which server should handle each request. The authors created a way to predict how long each step of processing takes and balance the work among servers while keeping delays low. Their method, LYREO, also learns faster from past results by turning delayed feedback into quick lessons. Tests show this approach reduces waiting times and balances workload better than other methods.
What this means in practice
- •For cloud infrastructure teams: Improve how requests for AI language models are distributed across edge servers to lower response times and avoid overload.
- •For content delivery network operators: Enhance distributed server load balancing for latency-sensitive AI services handling diverse and dynamic user requests.
Authors
Zhen Li, Jun Cai, Haoran Gao, An Li, Tan Li
Abstract
Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference, motivating the deployment of LLMs across distributed edge servers. However, heterogeneous communication and computing capabilities, together with dynamically evolving inference states, make the edge server selection for each incoming request time-varying and tightly coupled across slots. In this paper, we investigate an online request scheduling framework for edge LLM inference that jointly minimizes long-term average end-to-end latency and regulates workload distribution across heterogeneous edge servers. Two main challenges arise in this context. First, conventional latency models cannot accurately capture the fine-grained dynamics of multi-stage LLM execution. Second, the latency consequence of a scheduling decision is observed only after request completion, making immediate decision evaluation difficult. To address these challenges, we develop a cross-slot inference model that captures transmission, prefill, iteration-level decoding, and key-value (KV) cache evolution for each diverse request, and characterize server workload through a KV cache memory-time consumption metric. We propose the LYREO approach that transforms the long-term load-balancing constraint via Lyapunov optimization and employs reward redistribution with sequencebased return prediction to convert delayed outcomes into timely learning signals for earlier decisions. Simulations under various configurations demonstrate that LYREO consistently achieves lower latency and more balanced load distribution than representative learning-based and heuristic baseline schemes.