HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving
2026-08-03 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors looked at how to better run a single computer GPU for models that handle text, speech, and images all at once. They found that current systems don’t efficiently manage how different tasks share GPU time, which causes delays. They created HorizonServe, a system that smartly schedules and shares GPU resources to meet varying speed goals for different tasks. This solution made processing faster and more reliable across tests on different GPUs and workloads.
Omni modelsGPU schedulingService-level objective (SLO)Multimodal modelsLatencyResource allocationFirst-response timeStreaming multiprocessor (SM)Workload management
Authors
Yuning Zhang, Dong Yuan
Abstract
Omni models unify text, speech, image, and multimodal reasoning in a single serving backend, but this unified deployment exposes a new scheduling problem. Requests with different output modalities may share an initial multimodal backbone and then diverge into downstream generation stages, creating heterogeneous first-response metrics and service-level objective (SLO) targets on the same GPU. Existing large language model (LLM) and multimodal serving systems mainly optimize token progress or input-side processing, and they do not jointly control temporal sharing in the shared stage and spatial sharing among co-running stages. This paper presents HorizonServe, a single-GPU omni-model serving system that coordinates request admission and GPU allocation under heterogeneous SLOs. HorizonServe profiles per-class first-response latency, protects requests with limited slack, rotates shared-stage opportunities across execution paths, and throttles the shared-stage streaming multiprocessor (SM) allocation when downstream stages are active. Across three omni-model workloads and two GPU platforms, HorizonServe improves SLO attainment by up to 4.9$\times$ in arrival-rate sweeps and 7.0$\times$ under downstream-heavy traffic, and reduces per-class first-response latency by 38.4--63.7\%.