Shared caching speeds up 27B inference across GPU replicas
Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Running large language models on multiple GPUs can be slow if they each have to start fresh when handling parts of a task. The authors studied a way to share cached data between two GPUs running a big 27-billion-parameter language model to avoid repeating work. They found a bug causing errors and fixed it, showing that sharing cache cuts the delay to produce the first output token from tens of seconds to under a second. Their tests also showed when this shared caching helped performance and when it did not.
What this means in practice
- •For machine learning engineers: Speed up large language model inference across GPU replicas by sharing cached data, reducing delay in generating first output tokens significantly.
- •For cloud service operators: Improve efficiency of multi-GPU language model deployment using shared memory caching to reduce redundant computations and latency.
Authors
Frank Li
Abstract
Shared host-memory caching can avoid repeated prefill when a request moves between inference replicas. Its usefulness depends on both correct state transfer and lost prefix locality. We study two single-GPU 27B vLLM replicas sharing a 256 GiB LMCache pool. After adopting an existing packed-page patch, we isolate a raw-pointer fallback that omits the dependency on the current CUDA stream. Controlled byte tests fail under an imposed delay and pass when the dependency is restored; the existing mixed allocator provides a working deployment path. Full-pool allocation checks and service regression complete the validation. A four-block OFF-ON-ON-OFF comparison contains 768 measured requests within two block pairs. Median cross-replica time to first content token falls from 31.715 to 0.605 seconds at 128k input and from 92.047 to 0.790 seconds at 256k. Six-turn synthetic sessions alternating replicas improve by approximately 35% and 45% at initial contexts of 32k and 128k, while fixed placement shows little benefit. This engineering case study identifies practical validation steps and the locality conditions in which shared caching pays off.