Roomie: Interference-Aware Colocation for Efficient Model Serving
Distributed, Parallel, and Cluster ComputingOperating Systems
Summary
The authors address the problem of running multiple deep learning models on the same GPU without slowing each other down too much. They introduce Roomie, a system that predicts when different model computations will interfere with each other at a very detailed level. By using smart predictions and clever model placement, Roomie helps keep the processing fast and reliable. Tests show it reduces delays by up to three times compared to existing methods, working well both in big data centers and smaller edge devices.
Authors
Youssouph Faye, Francescomaria Faticanti, Shubham Jain, Francesco Bronzino
Abstract
As demand for DNN inference grows, GPU capacity is increasingly oversubscribed, forcing operators to colocate multiple models on the same device in both cloud and edge deployments. Whether colocation succeeds or violates SLOs depends on the temporal overlap of kernels from concurrently executing models -- an effect that existing serving systems either ignore or approximate using aggregate resource profiles that fail to capture temporal dynamics. This paper presents Roomie, a model serving orchestration architecture that predicts and avoids kernel-level interference between colocated DNNs. Roomie decouples offline kernel profiling from online interference prediction. It uses profiling only to extract per-kernel resource configurations, and predicts interference with an occupancy-based analytical model immune to profiler-induced timing distortion. A pairwise greedy heuristic then approximates multi-model interference in polynomial rather than exponential time, and an online placement algorithm then uses these estimates to assign each incoming model to the GPU that minimizes predicted slowdown. Our experimental evaluation compares Roomie against state-of-the-art solutions across both cloud-grade server clusters and embedded edge devices, demonstrating that Roomie reduces SLO violations (i.e., inference latency) by up to 3x, while maintaining comparable, and in many cases superior, goodput relative to existing approaches.