Ember improves GPU memory use to speed oversubscribed processing
Ember: Making Unified Virtual Memory Practical for Discrete-Memory GPUs
Operating SystemsDistributed, Parallel, and Cluster Computing
Summary
When computers with powerful graphics processors run out of fast memory, they slow down because data has to travel from slower memory. Ember is a system that smartly picks which data to keep on the fast graphics memory and which to keep on slower memory without needing special hints or changes to programs. It watches which data is used most often and moves just that data to the fast memory, making programs much faster than the default system. Ember works on existing GPUs and runs programs without changing them, greatly improving speed especially when the data size is bigger than the GPU memory.
What this means in practice
- •For machine learning engineers: Run large AI model workloads on GPUs with limited memory by automatically managing data placement to speed up processing without manual tuning.
- •For graphics software developers: Improve performance of graphics applications on GPUs by dynamically moving important data to GPU memory while oversubscribing it without application changes.
Authors
Senad-Leandro Lemes Galera, Michail Georgoulakis Misegiannis, Ferdinand Gruber, Jana Giceva, Dimitrios Stavrakakis
Abstract
GPU working sets have outgrown GPU memory, so applications must oversubscribe it, keeping only hot data on the GPU. NVIDIA's Unified Virtual Memory (UVM) supports oversubscription, but its performance degrades once data exceeds the GPU's memory. This stems from its fault-driven data placement, wasting interconnect bandwidth, and underutilizing the GPU. This raises the question: can UVM oversubscription become practical on stock GPUs, exploiting GPU memory speed and minimizing wasted interconnect bandwidth, without placement hints, profiling runs, or application changes? To this end, we present Ember, a UVM management framework for oversubscribed workloads. Ember introduces a hybrid address space; allocations start as mapped into host memory, omitting data migration. Ember's counter-driven region promotion uses GPU's hardware counters to fetch hot regions to the GPU, its decay-based eager reclamation proactively evicts cold regions in batches, while via online region re-placement, Ember tracks the hot set at runtime. We implement Ember within NVIDIA's open-source UVM driver for unmodified applications and as a userspace variant on stock CUDA. At 1.5x oversubscription, it is 6.8-10.6x faster than UVM on six applications across three PCIe GPUs, and generates LLM tokens 4.7-16x faster at sizes where manual placement cannot run at all. Without a profiling run, it beats a profile-guided alternative by 2.1x on kernel time.