LMEdge: QoS-Aware LLM Inference Orchestration on Edge Clusters
Distributed, Parallel, and Cluster Computing
Summary
The authors created LMEdge, a system that helps run large language models (LLMs) efficiently on different edge devices like phones or small servers. Their system decides which version of the model to use and where to run it, making sure responses are fast without losing accuracy and while respecting device limits. They use machine learning to predict how well each setup will perform and a smart method to quickly pick the best option. Testing LMEdge showed it makes LLM services quicker and better at using resources compared to other methods.
Authors
Reza Farahani, Zoha Azimi, Mario Colosi, Schahram Dustdar
Abstract
Large language model (LLM) services increasingly operate on edge infrastructure, enabling low-latency and privacy-preserving AI services. However, efficiently serving LLM requests across heterogeneous and resource-constrained edge devices require orchestration mechanisms that jointly determine model configuration (family, size, and quantization level) and execution placement while satisfying user- and system-level quality of service (QoS) requirements. This paper introduces LMEdge, a QoS-aware orchestration service that dynamically makes these decisions across heterogeneous edge devices. We formulate the problem as a binary integer linear programming (BILP) optimization that minimizes response time under accuracy, network, and resource constraints. To enable scalable online scheduling, we employ five lightweight machine learning (ML) models to predict query-specific latency, accuracy, resource usage, and response size for each model-size-quantization-device combination, and design a lightweight heuristic that approximates the BILP solution. We collect a comprehensive benchmarking dataset of over 59000 rows to train models and support reproducibility. Evaluation on a Kubernetes-based edge testbed with 57 instances and diverse query categories shows that LMEdge reduces latency, preserves accuracy, improves resource utilization, and increases serving ratio compared to two baselines.