Physical limits in chips affect GPU kernel speeds and optimization
Kernel Performance Engineering under Physical Non-Idealities
Hardware Architecture
Summary
Physical imperfections in how computer chips are made can slow down important computer tasks, even if software tries to hide those flaws. The authors looked at a special way GPUs handle memory and found that usual safety rules can limit how well tasks run. They created a better grouping method that works around these limits, speeding up some tasks by over 5%. They also found that using less power can let the chip run faster, and similar effects happen in other specialized chips, showing the need to consider real hardware limits when improving software.
What this means in practice
- •For gpu software developers: Improve GPU kernel speed by using wave-aware clustering techniques that respect hardware resource limits revealed by physical non-idealities.
- •For chip performance engineers: Optimize power and frequency management strategies by linking reduced data movement to higher GPU frequencies under thermal limits, increasing throughput.
Authors
Supun Kuruppu, Pranav Dangi, Wei Li, Zhenyu Bai, Tulika Mitra
Abstract
Physical non-idealities in integrated circuit (IC) design and fabrication are concealed by conventional hardware abstractions, yet they can limit kernel performance and obscure optimization opportunities. Using distributed shared memory (DSMEM) on NVIDIA H100 GPUs as a case study, we show how floorsweeping restricts hardware resources available for clustered execution, potentially offsetting the benefits of reduced data movement enabled by DSMEM. Guided by these insights, we develop a wave-aware clustering strategy that accounts for these resource constraints to improve kernel performance. Evaluations using a global-load-and-broadcast microbenchmark, GEMM, and the Hopper version of FlashAttention 4 demonstrate the performance implications of these constraints, with wave-aware clustering improving FlashAttention 4 throughput by up to 5.30% over its baseline. We further show that, under thermal design power (TDP) limits, reduced data movement enables higher Streaming Multiprocessor (SM) frequencies through Dynamic Voltage and Frequency Scaling (DVFS), explaining otherwise unexpected throughput improvements. Extending this analysis beyond GPUs, we show that power constraints also limit throughput scaling on Tenstorrent's Wormhole accelerator, highlighting the broader importance of accounting for physical non-idealities when analyzing and optimizing kernel performance.