Gpu sharing predicts operator interference to improve inference speed
PairSight: Characterizing and Predicting Operator-Level Interference in GPU Sharing for DNN Inference
Performance
Summary
Running deep learning tasks on GPUs is expensive, and usually a single task doesn't use all the GPU's power. Putting multiple tasks together can use the GPU better, but they can slow each other down unpredictably. The authors studied how these slowdowns happen at a fine level, looking at how individual parts of tasks interfere on shared hardware resources. They created a tool called PairSight that predicts how much tasks will slow each other down, helping to run multiple tasks efficiently.
What this means in practice
- •For gpu system engineers: Optimize GPUs running multiple DNN tasks by accurately predicting task slowdowns before co-locating workloads.
- •For cloud platform operators: Improve scheduling of AI inference services by estimating interference between tasks sharing GPUs to boost utilization.
Authors
Xiao Liu, Yujia Fu, Yue Yu, Dan Huang
Abstract
Serving DNN inference is costly, yet a single task seldom saturates both the compute units and the memory bandwidth of a GPU. GPU sharing raises utilization by co-locating several tasks on one GPU, at the cost of interference among them. Existing systems estimate this interference with binary compute-memory rules, standalone scaling curves, or model-level predictors. However, these estimates rest on a limited understanding of interference. They leave the software and hardware properties of co-located operators largely unexplored and give no slowdown prediction that is both interpretable and accurate. We conduct an operator-level study of interference under SM partitioning and use hardware counters to trace it to contention on the shared memory path, where the slowdown depends jointly on both co-located operators. Based on these findings, we propose PairSight, a hardware-aware interference predictor that learns how the hardware behaviors of co-located tasks interact and predicts the latency of each task before co-location. On six DNN models including LLMs, PairSight predicts latency under sharing with 4.3% mean error, against 19.0% for a compute-memory intensity rule, and adapts to unseen models and inputs with a few samples.