GPU sharing method predicts interference to ensure latency guarantees
Mosaic: GPU Sharing with Latency Guarantees through Kernel-Level Interference Prediction
Distributed, Parallel, and Cluster Computing
Summary
GPUs are heavily used for artificial intelligence but are often not fully used, so people try to run multiple tasks on the same GPU. This can cause delays for tasks that need quick responses because they interfere with each other. The authors created Mosaic, a new way to predict this interference more accurately by looking at how different parts of the GPU interact. They use this prediction to schedule tasks smartly, making sure important tasks finish on time while still using the GPU efficiently.
What this means in practice
- •For cloud infrastructure teams: Manage multi-tenant GPU clusters by predicting and controlling task interference to keep latency-sensitive AI workloads on schedule.
- •For real-time ai application developers: Schedule GPU workloads to guarantee response times while maximizing throughput for latency-critical AI services.
Authors
Foteini Strati, Ethan Graham, Leo Stephan, Paul Elvinger, Ana Klimovic
Abstract
GPUs are increasingly in demand for AI workloads, yet often remain substantially underutilized, motivating workload colocation. However, colocation introduces interference that can degrade latency-critical workloads. Existing approaches mitigate interference using either heuristic-based scheduling or interference predictors. Heuristics rely on coarse-grained metrics that overlook important interference sources, while predictors often depend on simulator-specific or similarly coarse metrics. However, GPU interference is complex and arises from multiple mechanisms, including thread-block placement, memory hierarchy contention, and intra-SM resource contention. We present Mosaic, a kernel-level interference predictor that explicitly models these mechanisms using a combination of analytical and lightweight learned models. Across four GPU architectures, Mosaic reduces prediction error by up to an order of magnitude compared to prior predictors. We integrate Mosaic into a scheduler, MosaicSched, that performs online kernel admission control and selects between full-GPU colocation and SM partitioning to maximize best-effort throughput while satisfying latency SLOs. Across all workloads, MosaicSched keeps the p99 latency below or very close to the target SLO.