Multi gpu apps run faster without cpu control overhead
Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in Charge
Distributed, Parallel, and Cluster Computing
Summary
Normally, computers with many graphics cards (GPUs) rely on the main processor (CPU) to manage work and communication between GPUs, which causes delays. The authors propose letting GPUs handle all communication tasks themselves after starting, without CPU involvement. They use existing techniques to enable this autonomous GPU communication, reducing the waiting time for messages between GPUs. Their tests show big speed improvements for common math problems solved across multiple GPUs. This approach can help programs that need GPUs to talk to each other often run faster.
What this means in practice
- •For high performance computing teams: Run iterative linear algebra solvers faster on HPC clusters by eliminating CPU involvement in GPU peer communication.
- •For machine learning infrastructure teams: Improve multi-GPU training efficiency by reducing communication overhead without relying on CPU coordination.
Authors
Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, Mohamed Wahib, Didem Unat
Abstract
This paper proposes a fully autonomous execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch. In a typical multi-GPU application, the host serves as the orchestrator of execution by directly launching kernels, issuing communication calls, and acting as a synchronizer for devices. We argue that this orchestration, or control flow path, causes undue overhead and can be delegated entirely to devices to improve performance in applications that require communication among peers. For the proposed CPU-free execution model, we leverage existing techniques such as persistent kernels, thread block specialization, device-side barriers, and device-initiated communication routines to write fully autonomous multi-GPU code and achieve significantly reduced communication overheads. We demonstrate our proposed model on two broadly used iterative solvers, 2D/3D Jacobi stencil and Conjugate Gradient(CG). Compared to the CPU-controlled baselines, the CPU-free model can improve 3D stencil communication latency by 58.8% and provide a 1.63x speedup for CG on 8 NVIDIA A100 GPUs. The project code is available at https://github.com/ParCoreLab/CPU-Free-model.