CEMA-Router improves routing for language models across cloud and edge
CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud-Edge Systems
Distributed, Parallel, and Cluster Computing
Summary
Running large language models on various devices in the cloud and edge is tricky because these devices have different speeds, costs, energies, and model qualities. The authors created CEMA-Router, a tool that looks at many factors together to decide how to best send groups of requests to different devices. By using smart search methods and real-time info about devices and workloads, CEMA-Router makes responses faster, cheaper, and more energy-efficient. Tests show it cuts energy use and delays substantially compared to current methods.
What this means in practice
- •For cloud infrastructure teams: Optimize routing of language model tasks across cloud-edge resources to save energy and reduce costs without sacrificing response quality.
- •For mobile app backend developers: Improve user request handling by efficiently assigning inference requests to devices with varying capabilities for faster, more reliable service.
Authors
Xijia Wang, TianZhang He, Mohammad Goudarzi
Abstract
Large language models (LLM) are increasingly integrated into pervasive applications spanning heterogeneous cloud-edge resources. These heterogeneous resources differ in computational capacity, energy efficiency, network conditions, and deployed model quality, creating trade-offs among response quality, monetary cost, energy consumption, and batch completion time when routing inference requests. Existing LLM routers typically make per-query routing decisions based primarily on model quality and token cost, whereas cloud-edge schedulers often overlook quality differences among deployed models. Consequently, these objectives remain insufficiently coordinated. We present CEMA-Router, an infrastructure-aware batch router for LLM inference in pervasive cloud-edge systems. CEMA-Router formulates batch routing as a multi-objective search problem that jointly minimizes model-quality loss, monetary cost, energy consumption, and straggler-dominated batch completion time. To efficiently explore the search space, it employs a routing-specific memetic algorithm, combining largest-request-first, load-aware seeding with probabilistic, evaluator-guided Lamarckian repair. Routing decisions are informed by deployment-specific node profiles, dynamic backlog estimates, and model-quality signals. We further implement CEMA-Router as an end-to-end system comprising a request manager, routing service, and profiler, supporting both real cloud-edge deployment and trace-driven simulation. We evaluate the system using RouterBench traces and GSM8K and MMLU workloads on a heterogeneous cloud-edge testbed. Compared with state-of-the-art methods, CEMA-Router reduces the weighted objective by up to 86% for GSM8K and 80% for MMLU, while reducing energy consumption by up to 33% and 19% and batch completion time by up to 27% and 30%, respectively.