Edge and near-edge hardware show trade-offs for running large language models
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
Running big language models on different devices near users involves trade-offs between speed, accuracy, energy use, and how big the models are. The paper compares a small edge device, a server with or without a GPU, and cloud service for answering questions. They find that GPU servers are fastest, the edge device uses the least energy, and CPU-only servers are slower and less efficient. They also show that model size alone doesn't predict performance or speed. This helps decide where to place language models for the best experience.
What this means in practice
- •For cloud service operators: Optimize placement of language model inference workloads between edge, near-edge, and cloud servers to balance latency and energy use for interactive services.
- •For mobile device engineers: Design energy-efficient on-device language model inference using edge hardware like Jetson Orin while accounting for accuracy and latency trade-offs.
Authors
Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos
Abstract
Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.