Die scaling causes GPU resource asymmetry that impacts performance
Dissecting How Die Scaling Breaks GPU Fine-grained Scheduling
Hardware ArchitectureDistributed, Parallel, and Cluster Computing
Summary
Modern GPUs have parts that aren’t exactly the same size or shape inside, due to how the chips are made smaller over time. This creates differences in how fast different parts of the GPU can talk to memory and work together. The authors show that if software doesn’t take these differences into account when giving work to the GPU parts, performance can vary a lot. They created a way to find out those differences for each chip and changed existing scheduling so GPUs can work more efficiently by assigning tasks to the best parts. This leads to faster performance for common GPU uses, like running machine learning models.
What this means in practice
- •For gpu system architects: Improve GPU task schedulers by adapting them to physical chip differences for steadier performance across GPU models.
- •For machine learning engineers: Boost efficiency of running multiple AI models on shared GPUs by scheduling tasks on hardware parts with better memory access.
Authors
Xiaoze Fan, Jianhao Wang, Weihao Cui, Han Zhao, Zhuobin Huang, Yangjie Zhou, Yuxian Qiu, Shixuan Sun, Bingsheng He, Quan Chen, Minyi Guo
Abstract
Modern GPUs are no longer physically symmetric. Die scaling leads to both manufacturing-driven floorsweeping and cache and memory partitioning. The former creates chip-specific compute topologies, while the latter causes non-uniform memory access. These asymmetries are substantial. Topology-oblivious compute unit allocation can lead to up to 1.33x performance variation, while remote accesses increase HBM latency by up to 67% and nearly double L2 latency. However, these asymmetries are hidden behind the GPU's logical resource abstractions and can vary across chips. We develop lightweight characterization methods to uncover per-chip compute topology and memory affinity. We then use the discovered information to make existing fine-grained scheduling asymmetry-aware, considering not only how many resources are allocated but also which physical resources are assigned. Across full-GPU kernel execution, intra-application multiplexing, and inter-application co-location, asymmetry-aware scheduling improves mainstream kernels by up to 1.22x, multiplexed LLM inference by up to 14.3%, and avoids up to 1.33x performance variation.