GAUSS simulates large language model serving for better latency predictions
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Artificial Intelligence
Summary
Large language models like those that generate text have complex workloads that are hard to predict using simple throughput numbers. The authors developed GAUSS, a simulator that can model different phases of model responses and how incoming requests queue up, capturing not only average speeds but also the delays users might see. Unlike methods that replay past data, GAUSS can explore new setups, hardware, and scheduling strategies that haven’t been used before. This helps operators understand and plan how to meet response time goals under varying workloads.
What this means in practice
- •For infrastructure operators: Predict how different hardware, workloads, and routing policies affect inference latency and throughput to meet service-level goals.
- •For cloud service architects: Evaluate scheduling algorithms and traffic segmentation strategies for deploying large language model inference at scale.
Authors
Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis, Anjaly Parayil, Dhruv Deshmukh, Victor Ruehle
Abstract
Large language model (LLM) inference has become an operational workload that fleets must provision, monitor, and bill against latency service-level objectives (SLOs). A single throughput number cannot answer the questions operators actually face, because generative serving couples two phases with very different service characteristics, continuous batching reshapes the effective service process, and user-visible tail latency can degrade long before aggregate throughput saturates. We present GAUSS, a workload-aware analytical and Monte-Carlo simulator that predicts the full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT), together with the sustainable request rate, for a given workload, engine, and accelerator. GAUSS pairs calibrated, phase-specific latency surfaces for prefill and decode with a Lindley-recurrence model of queueing under stochastic arrivals and estimates performance by ensemble Monte-Carlo sampling rather than by replaying a single trace. This lets GAUSS reason about workloads, routing policies, and hardware that have never been deployed, and to capture the latency tails that trace-driven simulators systematically under-sample. We describe the model and show how it supports operating-point selection, traffic segmentation, and shortest-remaining-processing-time scheduling capacity questions that trace-driven simulators cannot address.