Triple overlap boosts multi-GPU Krylov solver performance at scale
Triple Overlap in Pipelined Krylov Methods on Multi-GPU Systems
Distributed, Parallel, and Cluster Computing
Summary
Solving large scientific problems on many GPUs can be slowed down by communication delays. The authors show that overlapping three key tasks—computation, collective communication, and point-to-point communication—at the same time speeds up well-known Krylov solvers. They tested their approach on powerful supercomputers and found it performs better than existing methods, especially as the number of GPUs increases. This advance helps run important scientific simulations faster.
What this means in practice
- •For high-performance computing engineers: Speed up large-scale linear solver runs in supercomputing environments by improved overlapping of compute and communication across multiple GPUs.
- •For simulation software developers: Enhance scientific simulation software performance by integrating triple-overlap pipelined Krylov solvers optimized for multicore and multi-GPU deployments.
Authors
Sinan Ekmekçibaşı, James D. Trotter, Johannes Langguth, Xing Cai, Didem Unat
Abstract
We introduce a novel triple-overlap implementation of the pipelined preconditioned conjugate gradient (Pipe-PCG) and pipelined stabilized biconjugate gradient (Pipe-BiCGStab) solvers, methods widely used in scientific applications. By leveraging a collective communication library, our approach extends conventional computation-communication overlap to concurrently execute collective operations, irregular P2P communication, and compute kernels, enabling computation-communication-communication overlap across multiple GPUs. We evaluate the design on the MareNostrum 5 (H100) and Leonardo (A100) supercomputers, comparing against PETSc and the state-of-the-art two-stream NCCL implementation using SuiteSparse matrices and three massive instances scaled up to 512 GPUs. The three-stream design outperforms the two-stream baseline on every instance, with the margin widening from 2-6% at 4 GPUs to 18-47% at 64 GPUs. At 64 GPUs, the geometric-mean speedup over the two-stream baseline is 1.33x on MareNostrum 5 and 1.23x on Leonardo for Pipe-PCG, and reaches 1.30x for Pipe-BiCGStab. In large-scale experiments, we observed a speedup up to 35% in Pipe-PCG and up to 33% in Pipe-BiCGStab compared to the two-stream design. For PETSc, the margin is even wider, up to 3.27x speedup in both Pipe-PCG and Pipe-BiCGStab. Related code for the proposed method is available online: https://github.com/ParCoreLab/aCG