Entwine improves GPU tile computation and communication for faster results
Entwine: Coordinating Tiled Computation and Fine-Grained Communication across GPUs
Distributed, Parallel, and Cluster ComputingHardware Architecture
Summary
Computers use GPUs to handle big mathematical tasks by splitting them into smaller parts called tiles. These tiles finish at different times, which can cause delays when passing data around between GPUs. The paper’s authors designed Entwine to better coordinate when and how these tiles are processed and communicated. By scheduling tile work more smoothly and controlling resource sharing, Entwine speeds up overall computation. Tests show it outperforms current leading methods on big language model tasks.
What this means in practice
- •For gpu system developers: Improve coordination of tile computations and communication in multi-GPU setups to reduce runtime in high-performance workloads.
- •For machine learning engineers: Accelerate tensor-parallel large language model training by optimizing GPU resource allocation and data exchange patterns.
Authors
Kai Ma, Quanfeng Lv, Jingguo Ge, Bowei Dai, Kefan Ruan
Abstract
Modern high-performance GPU computations partition tensors into tiles to exploit data reuse and parallelism. Individual tile computations complete earlier than the full tensor computation, creating opportunities to overlap computation and communication. However, a mismatch between computation and communication progress can limit these opportunities. Communication stalls when no data is ready, and may lag when data arrives in bursts. Communication can also slow computation by consuming shared resources, offsetting the benefits of overlap. We present Entwine, which coordinates tile computation order, fine-grained communication, and SM resource allocation to minimize overall completion time. Entwine reorders tile computation to produce data for communication at a more regular pace. Entwine couples this schedule with fine-grained SM-based communication to process tile results with low latency and low overhead. Since the communication kernel also consumes SM resources, Entwine coordinates their allocation to balance communication progress against computation slowdown. Across representative tensor-parallel LLM workloads, Entwine achieves a geomean speedup of 1.232x (up to 1.433x) over cuBLAS+NCCL, and outperforms state-of-the-art overlap baselines by 3.1-9.8% in geomean. We will open-source our implementation upon publication.