Reinforcement learning improves scheduling on heterogenous accelerators for real time dnn inference
Hard and Soft Real-Time DNN Inference Scheduling on Heterogeneous Multi-Accelerator Systems via Reinforcement Learning
Hardware ArchitectureDistributed, Parallel, and Cluster Computing
Summary
Scheduling tasks in systems with different kinds of AI accelerators is tricky, especially when some need strict timing and others can be more flexible. The authors created a new system that uses reinforcement learning to decide which tasks to prioritize and where to run them, while also making sure the most urgent tasks always meet their deadlines. They introduced a way to check schedules on the fly, ensuring that the system stays stable even with complex task orders. Tests showed their method keeps hard timing tasks on track while improving overall task quality and hardware use.
What this means in practice
- •For embedded system engineers: Optimize scheduling on devices with multiple AI accelerators to meet strict and flexible timing needs for real-time neural network tasks.
- •For cloud infrastructure managers: Improve utilization and quality of service in server clusters running mixed real-time AI workloads on various accelerator types.
Authors
Francesco Giulio Blanco, Enrico Russo, Giuseppe Ascia, Davide Patti, Maurizio Palesi
Abstract
We present a scheduling framework for heterogeneous multi-accelerator systems executing both hard real-time and soft real-time QoS-aware DNN inference workloads. The framework combines an online reinforcement learning (RL) scheduler with a deterministic runtime safety layer. The RL scheduler dynamically generates candidate schedules by jointly selecting sub-job priorities and accelerator assignments, while runtime certification guarantees that all HRT deadlines remain satisfied independently of the learned policy. To enable runtime certification in heterogeneous precedence-constrained systems, we introduce a schedulability-preserving prefix replacement criterion based on machine-wise cuts. The proposed criterion guarantees that runtime-generated schedule prefixes preserve the existence of a feasible continuation without requiring complete finish-time dominance with respect to a feasible reference schedule. Experimental results on heterogeneous DNN accelerator platforms show that the proposed framework preserves HRT schedulability while improving QoS-aware deadline satisfaction, client-level service quality, and accelerator utilization compared to state-of-the-art scheduling approaches.