Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
2026-06-08 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
AI summaryⓘ
The authors studied how to speed up training large machine learning models on multiple GPUs by doing computation and communication at the same time instead of one after the other. They use two techniques to manage GPU resources: controlling how much shared memory each computation task uses and giving communication tasks higher priority so they don’t get blocked. Their experiments on various GPUs show that these changes can reduce total training time by up to about 25%, all without needing to change existing software libraries. This means faster training by better overlapping work on the GPUs.
distributed trainingmulti-GPUcommunication overheadcomputation-communication overlapshared memorykernel schedulingNVIDIA A100AMD MI250Xcollective communicationexecution time optimization
Authors
Minyu Cui, Miquel Pericas
Abstract
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, communication overhead has become a dominant bottleneck in multi-GPU training, particularly when computation and communication are executed sequentially. This work explores concurrent execution of computation and collective communication using two portable runtime controls: shared-memory-driven occupancy shaping for computation kernels and elevated scheduling priority for communication kernels. Our approach regulates computation-kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress. In addition, assigning higher priority to communication streams ensures steady communication progress once resources become available. Experiments on NVIDIA A40, A100, H100, and AMD MI250X GPUs demonstrate that the proposed method enables effective computation-communication overlap and reduces total execution time by up to 25.5 percent, without modifying vendor libraries or kernel implementations.