Scheduling inference workloads reduces delays and memory costs

Queuing for Inference: Dynamic Memory Footprints and Costly Preemption

PerformanceNetworking and Internet Architecture

Summary

Inference tasks for transformer neural networks can have tricky memory needs and delays caused by long waits, partly because of how tasks interrupt each other. The authors created a new queuing model that takes these issues into account to better understand scheduling performance. They found that a popular scheduling method, the Gittins-index policy, sometimes causes lots of overhead due to interruptions. To fix this, they proposed a new method called Multi-Level-Priority-Threshold (MLPT), which limits interruptions but still helps short tasks finish quickly. Their work balances quick responses with keeping interruption costs low and offers a framework for scheduling in memory-limited inference systems.

What this means in practice

  • For data center operators: Optimize load balancing and scheduling of inference tasks to reduce delays and resource overhead in memory-constrained environments.
  • For edge device engineers: Improve scheduling strategies to manage limited memory efficiently for on-device transformer inference workloads.

Authors

Jiahui Ni, Vishrant Tripathi

Abstract

Modeling inference workloads for transformer-based neural networks poses unique challenges due to their dynamic memory footprints, head-of-line blocking caused by heavy-tailed request distributions, and memory-dependent preemption overhead arising from the need to load/unload cache state. In this paper, we introduce a new queuing model that captures the impacts of these features on system stability and scheduling design. We show that the Gittins-index policy, which minimizes mean response time in the standard M/G/1 setting, can incur substantial preemption overhead in our model. Motivated by this limitation, we propose the Multi-Level-Priority-Threshold (MLPT) policy, that limits preemption while preserving priority for short jobs. We characterize the mean response time under MLPT and provide empirical rules-of-thumb to solve the resulting threshold-optimization problem. Our key observation is that optimal scheduling is a balance between reducing the response times of shorter jobs and limiting the overhead incurred when preemption occurs. These results provide a theoretical framework for understanding scheduling in memory constrained queuing systems for inference tasks.