Balancing GPU parallelism to speed up large language model responses
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine LearningPerformance
Summary
Large language models need to produce answers quickly and handle many requests at once, but using many GPUs efficiently is tricky. The paper explains that inference has two parts: starting to generate the first word and continuing to produce following words. Different ways of splitting work across GPUs affect these speeds differently. The authors create a method to find the best way to split GPU work based on request types, speed goals, and context length. They find that settings favoring quick start times are different from those that maximize total throughput, especially as the text context grows longer.
What this means in practice
- •For cloud service operators: Configure multi-GPU cloud servers to balance quick initial responses and overall throughput based on workload types and latency requirements.
- •For machine learning infrastructure teams: Optimize GPU parallelism strategies to support long text contexts efficiently for language model serving.
Authors
Javad Mirzaei, Jeebak Mitra
Abstract
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.