GPU method speeds training of massive graph neural networks fourfold
GARDEN: Efficient GPU-Accelerated Billion-Scale GNN Training via Caching-Sampling-Batching Co-Design
Distributed, Parallel, and Cluster Computing
Summary
Training graph neural networks on very large graphs is hard because GPUs cannot hold all the required data at once. The authors developed a new system called GARDEN that smartly stores frequently used data in GPU memory, orders data sampling to speed up GPU processing, and groups training batches to make data transfers more efficient. This system speeds up training by up to over four times compared to previous methods, without losing accuracy. Their approach works on real-world graphs with billions of connections.
What this means in practice
- •For social network engineers: Train large-scale social network analysis models faster by efficiently handling feature and graph data on GPUs.
- •For recommendation system developers: Accelerate training of billion-edge recommendation graphs to improve model iteration speed and deployment cycles.
Authors
Qingrui Zhu, Lei Liu, Yushu Cai, Kai Sheng, Zhiwen Chen, Xin He
Abstract
Graph Neural Networks (GNNs) have shown impressive performance in diverse tasks such as social network analysis and recommendation systems. However, scaling their training to billion-edge graphs is hindered by the memory wall: GPU memory capacity falls short of the massive storage requirements for node features and topology. Sampling-based GNN training frameworks mitigate this by streaming subgraphs from CPU to GPU, but they incur prohibitive PCIe data transfers and non-coalesced memory accesses, resulting in bandwidth underutilization and low compute efficiency. We propose GARDEN, a GPU-accelerated sampling-based system that co-designs caching, sampling, and batching to eliminate these bottlenecks. GARDEN introduces three key innovations: (1) an Adaptive High-Fidelity Cache Allocator (AHFCA) that combines offline reuse profiling and online hotness tracking to dynamically cache features and adjacency lists within a fixed GPU memory budget, avoiding static allocation waste; (2) a Grouping-based Parallel Sampler that reorders stochastic neighborhood accesses(GPS) to maximize warp occupancy and hide latency on power-law graphs; and (3) a Super-batch Degree Classifier(SDC) that fuses mini-batches into degree-homogeneous groups, enabling cache-friendly coalesced transfers and high GPU utilization. Evaluations on real-world billion-scale graphs demonstrate that GARDEN achieves 4.54x, 1.27x, and 2.97x end-to-end training speedups over DGL-UVA, DUCATI, and FastGL, respectively, while maintaining model convergence and accuracy.