Desynchronizing MPI programs improves speed by balancing memory and compute use
Exploiting the Interplay of Compute- and Memory-Bound kernels in MPI Applications
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Some computer programs use many processors that usually work together step by step, trying to avoid waiting for data from others. The authors found that in some programs that switch between heavy computations and memory-heavy tasks, small delays in communication can actually help speed things up by easing competition for memory access. They showed this effect using a program that does image analysis and light communication, and confirmed it with experiments and simulations. This means letting processors run slightly out of sync can be a smart way to improve performance on certain computers.
What this means in practice
- •For parallel software developers: Optimize execution by allowing natural phase shifts to reduce memory bandwidth contention in compute- and memory-bound mixed workloads.
- •For high performance computing system administrators: Tune process placement and communication methods to improve throughput on ccNUMA architectures by balancing memory use across processes.
Authors
Ayesha Afzal, Krishna Manda, Georg Hager
Abstract
Parallel applications are often designed for synchronous, lock-step execution, treating communication stalls as performance hazards. Yet, in a communication-light application without frequent synchronization points that alternates between compute-bound memory-bound execution, an MPI communication stall can act as an unintentional relief on memory-bandwidth contention. We demonstrate this using a Parallel Optical Flow Solver, which combines a compute-bound Ray Tracing kernel with a memory-bound Optical Flow Solver kernel and negligible inter-process communication. This program shows considerable speedup via desynchronization and automatic overlap between compute- and memory-bound phases, showing that natural desynchronization is an architecture-aware optimization. An optimal speedup is achieved when the number of processes concurrently executing the memory-bound phase on a ccNUMA domain is near the bandwidth saturation point. We also show a case where reducing communication overhead using MPI asynchronous progress significantly degrades performance because it allows too many ranks to contend for memory bandwidth simultaneously. In order to study the dynamics under more controlled conditions, we develop a tunable dual-kernel microbenchmark, with which we show that significant application or system noise (natural or injected) is required to achieve full desynchronization. Finally, we also validate these results using a bandwidth-aware, model-based simulator.