Summary
NVIDIA’s Grace Blackwell system uses a unified memory approach that lets the CPU and GPU share memory address spaces, but accessing that memory can behave very differently depending on where the data physically lives. The authors studied how memory performance changes when data is stored in GPU-local memory versus system memory, and how techniques like prefetching and reuse affect speed. They found that simply having unified memory doesn't guarantee consistent speed, and actual performance depends on factors like memory placement, usage patterns, and capacity limits. This research provides detailed measurements and a model to help developers understand and optimize memory use on these systems.
What this means in practice
- •For gpu kernel developers: Optimize CUDA kernels by choosing memory placement strategies that better match access patterns to improve data bandwidth and reduce migration overhead on Grace Blackwell systems.
- •For high performance computing teams: Plan memory usage to avoid capacity oversubscription effects that degrade performance in unified memory systems by following this paper’s placement and reuse insights.
Abstract
NVIDIA Grace--Blackwell systems make CPU--GPU memory increasingly unified at the programming level, but unified addressability does not make the memory system uniform. A CUDA managed allocation on GB200 may be served from GPU-local HBM, accessed through host/system memory, demand-migrated, prefetched, reused after GPU access, or evicted and reloaded under capacity pressure. The problem is that these states share a programming interface while exercising different hardware paths and runtime mechanisms, making a single ``Unified Memory bandwidth'' number misleading. This paper presents a placement-controlled microbenchmark study of CUDA Unified Memory on NVIDIA GB200, using GH200 as a Grace--Hopper baseline and GB10/DGX Spark as a validation platform. Our methodology separates local HBM access, host/system-memory access, explicit movement, managed first touch, prefetch, reuse, access-pattern sensitivity, oversubscription, and CPU--GPU handoff, then summarizes the observations with a measurement-calibrated model of compute, access, migration, eviction, synchronization, and handoff costs. On GB200, a 1~GB STREAM-style triad reaches 3963~GB/s from local HBM but 304~GB/s through host-mapped CPU/system memory; for a 256~MB streaming kernel, cold CPU-initialized managed memory reaches 14.1~GB/s, while managed prefetch and reuse reach 4606~GB/s and 4504~GB/s. Below the capacity boundary, prefetched and reused managed memory remain above 4.6~TB/s, but at 125\% of CUDA-visible memory they fall to 968~GB/s and 1214~GB/s, showing that Grace--Blackwell unified memory must be understood through placement, access history, locality, reuse, and capacity pressure.