Composable CXL memory enables shared cache across Kubernetes nodes
Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models need fast access to cached information to respond quickly, but sharing this cache between computers is hard. The authors created a system that lets multiple computers running Kubernetes share a big slice of special memory called CXL. This shared memory acts like a common cache that all the computers can use, cutting down on repeated work and speeding up responses by up to 36 times. The system also cleverly organizes the cache so it doesn’t need extra servers to keep track of data. They tested their idea on a two-computer setup and showed sharing this memory adds almost no extra delay.
What this means in practice
- •For cloud infrastructure teams: Use composable CXL memory as a shared cache tier to speed up large language model serving across multiple Kubernetes nodes without external metadata services.
- •For ai platform engineers: Implement cross-node KV-cache reuse on distributed AI model deployments to reduce recompute and improve time to first token performance.
Authors
Hongjian Fan, Kevin Zhang, David Habinsky, Sean Dykstra
Abstract
We present a Kubernetes Dynamic Resource Allocation (DRA) driver that makes composable CXL memory a schedulable cluster resource, and evaluate the resulting shared-memory tier for cross-node KV-cache reuse in LLM serving. The driver composes CXL regions on demand, materializes them as DAX devices on each participating host, and injects them into pods under a single Container Device Interface (CDI) name so that pods on different nodes access the same physical region. A shared-memory connector for vLLM/llm-d uses that region as a KV-cache tier with a slot directory embedded inside the shared medium, which eliminates the need for an external metadata service. On a two-node cluster with a 512\,GiB CXL appliance and Qwen2.5-7B-Instruct, cross-node prefix reuse reduces TTFT by 5.5$\times$--36.6$\times$ at an external hit rate of 95.4--99.5\,\%, while node-local tiers (GPU prefix caching, CPU-DRAM offload) fall back to full recompute. The sharing gap, defined as the latency ratio between cross-node and same-node reuse, is 1--4\%, indicating that cross-node reuse incurs little additional latency relative to same-node reuse on our testbed. Both replicas run full engines; the study demonstrates memory disaggregation rather than prefill/decode disaggregation. We report this as a feasibility study rather than a performance evaluation.