Strategies for splitting large AI models across multiple GPUs explained
Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine LearningPerformance
Summary
Running big AI language models requires many graphics cards working together because one card is not enough. The paper explains different ways to split the work: by sharing computations across cards (tensor parallelism), by dividing tasks into stages (pipeline parallelism), or by combining both. The authors created a model that predicts how these choices affect speed and communication delays, especially in different parts of running the AI. Their work helps engineers decide the best way to use multiple GPUs efficiently for AI services.
What this means in practice
- •For data center operators: Plan GPU cluster setups to meet latency and throughput goals for AI language model serving by selecting optimal parallelism strategies.
- •For cloud infrastructure engineers: Optimize configurations for distributing large AI models across GPUs to reduce delays and maximize resource use during language model inference.
Authors
Javad Mirzaei, Jeebak Mitra
Abstract
Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly distributed across multiple GPUs using tensor parallelism (TP), pipeline parallelism (PP), or hybrid parallelism (HB). However, selecting the most effective parallelism strategy remains challenging due to complex interactions among computation, communication, pipeline utilization, sequence length, batch size, and model architecture. Existing approaches largely rely on empirical evaluation and provide limited analytical insight into the trade-offs among these strategies, particularly across the distinct prefill and decoding phases of inference. In this paper, we present a unified analytical framework for modeling distributed LLM inference under TP, PP, and HB. The framework decomposes end-to-end latency into computation, inter-GPU communication, and pipeline bubble overhead, and derives analytical models that capture TP collective communication, PP point-to-point communication, and pipeline utilization as functions of hardware, model, and workload characteristics. The model further characterizes the differing execution behavior of prefill and decoding, explaining why PP-oriented configurations favor compute-intensive prefill while TP-oriented configurations reduce decoding latency by eliminating pipeline bubbles. Experiments with modern LLMs on multi-GPU platforms validate the model and confirm the fundamental compute-communication trade-off across parallelism strategies. The framework provides practical guidance for parallelism selection, capacity planning, and optimization of future LLM serving systems.