Gpu performance improved by controlling prefetch timing and placement
SnakeCharmer: Orchestrating Utility, Timeliness, and Placement in GPU Prefetching
Hardware Architecture
Summary
GPUs can slow down when many parts wait for data from memory. Predicting what data will be needed next helps, but predictions only improve speed if the data arrives at the right time and place without causing problems. The authors introduce SnakeCharmer, a system that decides if, when, and where to load predicted data in GPUs to boost speed and efficiency. Tested on a model GPU, it showed clear improvements over earlier methods without needing much extra hardware.
What this means in practice
- •For gpu hardware designers: Implement SnakeCharmer's backend control to reduce stalls and improve GPU memory efficiency for better overall device performance.
- •For high performance computing teams: Enhance GPU workloads by adopting SnakeCharmer to optimize data prefetch timing and placement, boosting execution speed and energy use.
Authors
Yuhan Tang, Tiejun Li, Jianmin Zhang, An Hu, Guangyu Sun, Sheng Ma, Yiqun Lang, Hanqing Li, Siqing Fu
Abstract
GPUs rely on massive thread-level parallelism to tolerate long memory latency, yet they still suffer severe stalls when many warps simultaneously wait for data from the memory system. Address-prediction-based GPU prefetchers can mitigate such stalls, but accurate prediction alone does not guarantee performance improvement: a predicted prefetch becomes beneficial only if it is worth issuing, arrives at the right time, and is retained without harmful interference. Because these requirements interact, optimizing any single dimension in isolation can simply move the bottleneck to another. We capture these requirements with a predicted-prefetch-to-realized-benefit perspective and propose SnakeCharmer, a unified backend prefetch-control framework for address-prediction-based GPU prefetchers. SnakeCharmer separates address prediction from prefetch realization: the frontend predictor generates prefetch candidates, while the backend control path jointly controls utility, timeliness, and placement by determining whether each candidate should be issued, when it should be issued, and where the returned data should be retained. SnakeCharmer realizes this control path with zero-value-aware filtering for utility, hierarchical feedback-driven distance tuning for timeliness, and a register-assisted prefetch buffer for placement. We design and implement SnakeCharmer primarily on top of Snake, and adapt the same backend-control principles to APOGEE to demonstrate that SnakeCharmer's benefits are not limited to a single frontend predictor. On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 12.8%/20.8% over Snake and by 1.7%/3.2% over APOGEE, with modest hardware overhead.