Unified early-exit system cuts AI delays on edge devices
UniExit: A Unified Latency-Aware Early-Exit Runtime for Heterogeneous Edge AI
Distributed, Parallel, and Cluster Computing
Summary
AI models often take varying amounts of time to give answers, especially on small devices like smartphones or sensors. The authors created UniExit, a system that decides the best moment to stop the AI’s work early to save time based on how hard the problem is and the device’s current state. This system works for many types of AI models, including those for images and language, making AI faster and more efficient without losing much accuracy. It adapts in real time and works well on different hardware setups, reducing wasted work and saving energy.
What this means in practice
- •For edge device developers: Manage AI model execution on resource-limited devices to meet strict time budgets with less wasted computation.
- •For cloud-edge infrastructure teams: Coordinate AI tasks between edge and cloud processors to improve overall responsiveness and energy use in mixed workloads.
Authors
Yuhang Guan, Huanle Zhang, Xiaomin Ouyang, Zhidan Liu
Abstract
Edge applications increasingly combine CNN-based perception with edge-deployable Transformer-based language models, but deploying such heterogeneous workloads on resource-constrained devices is challenging because inference latency varies with both input difficulty and runtime hardware state. Early-exit inference reduces computation by terminating at intermediate stages, yet existing methods are often model-specific and rely on static latency profiles that become stale under dynamic edge conditions. This paper presents UniExit, a unified latency-aware early-exit runtime for heterogeneous edge AI. UniExit abstracts CNN blocks and Transformer layers as ordered stages and applies a common latency-aware routing policy while retaining model-specific quality mechanisms. It predicts the cost of reaching the next exit from runtime state and inference progress to make budget-aware continue-or-exit decisions. UniExit combines runtime-state-aware latency prediction, cache-based prediction reuse, trend-aware fallback, and CNN-specific exit stabilization to provide low-overhead and robust online control. We implement UniExit on multiple edge and edge-cloud platforms and evaluate it with representative CNN and Transformer-based language-model workloads. Across single-workload, concurrent heterogeneous, and edge-cloud settings, UniExit consistently improves deadline satisfaction and latency-accuracy trade-offs over existing progressive inference baselines, while reducing redundant exit-module computation and task-level energy consumption.