Elastic memory sharing boosts GPU performance in data centers
EMA: Elastic and Performance Transparent Memory Across GPUs
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
Large computer servers often use many GPUs that each have their own memory, but sometimes one GPU runs out of memory while others have extra. The authors developed EMA, a system that lets GPUs share memory easily and quickly, as if all the memory were local to each GPU. EMA also makes sure that lending GPUs do not lose performance because their memory can be taken back when needed. This helps programs run faster and handle bigger problems without needing extra hardware.
What this means in practice
- •For data center operators: Improve GPU utilization and throughput by enabling dynamic memory sharing across GPUs in multi-GPU servers.
- •For cloud service providers: Offer more flexible GPU-based inference services that can handle variable memory demands without reducing performance.$Commercial implications: Enables selling GPU cloud services that efficiently handle large, dynamic workloads like AI inference with better performance guarantees.
Authors
Yi Xu, Tian Xia, Ion Stoica
Abstract
Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2X capacity, and maintains latency similar to the static local baseline.