Scheduling inference workloads reduces delays and memory costs
Queuing for Inference: Dynamic Memory Footprints and Costly Preemption
Summary
Inference tasks for transformer neural networks can have tricky memory needs and delays caused by long waits, partly because of how tasks interrupt each other. The authors created a new queuing model that takes these issues into account to better understand scheduling performance. They found that a popular scheduling method, the Gittins-index policy, sometimes causes lots of overhead due to interruptions. To fix this, they proposed a new method called Multi-Level-Priority-Threshold (MLPT), which limits interruptions but still helps short tasks finish quickly. Their work balances quick responses with keeping interruption costs low and offers a framework for scheduling in memory-limited inference systems.
What this means in practice
- •For data center operators: Optimize load balancing and scheduling of inference tasks to reduce delays and resource overhead in memory-constrained environments.
- •For edge device engineers: Improve scheduling strategies to manage limited memory efficiently for on-device transformer inference workloads.