GKSM cuts GPU memory use by over half for serverless AI inference
gKSM: GPU Memory Deduplication at Driver Layer for Serverless LLM Inference
Operating Systems
Summary
Cloud providers running many AI chatbots on the same GPU waste a lot of memory by storing the same data multiple times. The authors created gKSM, a system that smartly finds and removes these duplicated data inside GPU memory without slowing things down. It works by having the CPU and GPU collaborate to detect duplicates and handle data safely, letting more AI chatbots run at once and boosting overall performance. Their tests show it can reduce memory needs by nearly 60% and more than double throughput under tight memory conditions.
What this means in practice
- •For cloud infrastructure engineers: Optimize GPU memory usage in hosting multiple AI model instances simultaneously to increase throughput and reduce costs in serverless platforms.
- •For gpu software developers: Incorporate GPU memory deduplication into driver-layer tools to enable efficient model weight sharing without native GPU copy-on-write support.
Authors
Shuheng Gao, Fan Guo, Yongkun Li, Ziye Yue, Yinlong Xu, John C. S. Lui
Abstract
In serverless LLM inference, cloud providers commonly locate multiple instances on a single GPU to improve utilization. However, each instance independently maintains its model weights and runtime state, making duplicate GPU memory the primary bottleneck. Memory deduplication is a widely-used technique for eliminating duplicate data in CPU systems, but extending it to GPU memory faces fundamental challenges. Effective GPU memory deduplication must avoid redundant data movement between CPU and GPU while preserving write isolation for shared pages without native copy-on-write (COW) support. This paper presents gKSM, a transparent GPU memory deduplication framework for serverless LLM inference. gKSM addresses these challenges through CPU-GPU collaborative hash management that offloads hash computation to GPU kernels, zero-copy model weight deduplication that eliminates duplication before model weight transfer, and a cross-privilege page fault handler that combines a lightweight kernel hook and CUDA Virtual Memory Management (VMM) remapping in the user space. gKSM also designs a stable page scanning algorithm and a reuse-aware placement policy in cluster scale. Experiments show that gKSM reduces GPU memory usage by 59.8% while maintaining high throughput compared to ServerlessLLM. Under memory constrained workloads, gKSM improves aggregate throughput by 2.15X and supports five instances concurrently. In cluster level simulation, gKSM achieves a service rate of 99.97%, compared to 78.2% for SLInfer and 26.9% for ServerlessLLM.