EdgePace improves edge device task timing with smarter memory use
EdgePace: Deadline- and Freshness-Aware LLM Scheduling on Unified-Memory Platforms
Performance
Summary
EdgePace helps devices like smartphones and robots run big language models and other tasks smoothly at the same time. Normally, these tasks share memory and can slow each other down, causing delays that matter in real-time situations. The authors designed EdgePace to watch how memory is used and smartly pause or schedule the language model work so other tasks meet their deadlines. This keeps results fresh and useful without losing much language model speed.
What this means in practice
- •For robotics engineers: Manage memory and computing tasks to keep sensor-driven decisions timely while using large language models for guidance.
- •For mobile device developers: Enable smartphones to run big language models and camera or sensor processing simultaneously without missing critical timing constraints.$Commercial implications: This paper enables product features that combine AI language processing and real-time sensing, supporting advanced smartphone capabilities sold to consumers.
Authors
Teng Mei, Chengxuan Pei, Weijun Wang, Jie Wu, Marco Canini, Yunming Xiao
Abstract
Large language models (LLMs) increasingly run on edge devices alongside tasks that continuously process sensor data. In many applications, these foreground tasks prepare observations as inputs to the LLM, whose outputs guide subsequent processing or actions. The observations must be processed by foreground deadlines, and LLM feedback must arrive while its source observations remain useful. Yet the CPU and accelerators share a unified-memory system, so LLM inference can delay foreground processing through memory-bandwidth contention even when compute resources remain available. We present EdgePace, a bandwidth-aware runtime that protects foreground deadlines while sustaining timely LLM feedback. EdgePace creates safe suspension points across prefill and decode, monitors foreground memory demand and stalls at low cost, and schedules inference according to contention risk, deadline slack, and output freshness. We evaluate EdgePace on a OnePlus 13 and three NVIDIA Jetson platforms using multi-camera processing and robotic target tracking. Relative to unmanaged inference, it reduces foreground deadline misses by 24.1%--96.1% while retaining 64.9%--98.5% of LLM throughput. All measured LLM outputs arrive within their task-defined freshness windows. On OnePlus 13, for example, camera deadline misses fall by 91.3% while 90.4% of LLM throughput is retained.