Papers for

cloud gpu infrastructure teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Agentic lLM serving improves speed by better KV cache eviction

Rethinking KV Cache Eviction for Agentic LLM Serving

Abstract: Key-value (KV) caches impose substantial memory pressure on large language model serving systems, requiring cache eviction when capacity is insufficient. Existing mechanisms typically evict KV from the ends of cached sequences to preserve prefix sharing across requests. However, agent workloads repeatedly reuse long histories across successive turns. Recomputing the KV cache for tokens near the end of a sequence is substantially more expensive than for earlier tokens because their attention computation involves longer histories. To address this problem, we propose preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix. We build Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management. Returning requests reuse the KV cache retained in GPU or CPU memory and recompute only missing segments while preserving reuse of the cached suffix. We implement Penelope in SGLang and evaluate it on three Qwen3 model sizes using real agent traces. Compared with SGLang, Penelope reduces mean time to first token by up to 60% and job completion time by up to 37% with a GPU-only cache, and by up to 76% and 60% with a two-tier GPU-CPU cache.

Fri 30 OctDistributed, Parallel, and Cluster Computing
The gist
Large language models use a key-value (KV) cache to remember past conversation tokens, but this cache can get full and slow down the system. Existing methods usually remove tokens from the end of the conversation history, which wastes time because recent tokens rely on all previous context for attention. The authors propose a new way to keep the end of the conversation intact and remove older tokens from the beginning instead. This approach, implemented in their system Penelope, reduces the delay before the first answer and overall response time by efficiently managing the cache on both GPUs and CPUs.
Open → 2611.00391v1

SlackSwap improves training speed with smarter GPU memory use

SlackSwap: Bandwidth-Aware Activation Swapping and Recomputation for Memory-Efficient DNN Training

Abstract: Swapping saved activations to host memory once enabled GPUs to train models that exceeded their device-memory capacity when compute throughput and host-link bandwidth were roughly balanced. For example, on a 2015 Titan X, PCIe bandwidth was sufficient to offload all VGG-16 activations during the forward pass. This balance no longer holds. From the Titan X to the H100, peak training throughput increased by 141x, whereas host-link bandwidth improved by only 3.9x. On an A100, the host link can hide transfers for only 11-38% of the activations in the transformer models we study. Existing swapping approaches, such as vDNN, SuperNeurons, and SwapAdvisor, offload tensors according to memory requirements without explicitly respecting the available link-time budget, resulting in transfer-induced stalls; block checkpointing, in contrast, leaves the host link underutilized. We present SlackSwap, which treats host-link time as an explicit resource budget. Using a single profiled iteration, SlackSwap allocates the forward-pass link budget to tensors that are most expensive to recompute, resolves the remaining memory deficit by recomputing the least expensive tensors, and schedules FIFO offloads and earliest-deadline-first prefetches without per-layer synchronization barriers. In trace-driven simulations of BERT-Large, GPT-2 XL, and a Llama-7B-style model on an A100, SlackSwap retains 96-100% of unlimited-memory GPU throughput while using half the activation memory, achieving 1.10-1.11x the throughput of the best baseline. Its plans are within 0.5% of a genetic search over 6,600-14,000 candidate plans, while requiring less than 0.4 seconds to generate.

Fri 9 OctDistributed, Parallel, and Cluster Computing
The gist
Training large AI models on GPUs can run into memory limits, forcing slow data swaps between GPU and host memory. The authors show that past swapping methods don’t perfectly balance data transfers and recomputations with the link speeds available today. They introduce SlackSwap, a smarter way to manage these swaps by prioritizing which data to transfer or recompute based on bandwidth constraints. This approach keeps training fast while using less GPU memory, nearly matching the speed of setups with unlimited memory.
Open → 2610.13003v1

TempoKV improves memory caching for faster large language model serving

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

Abstract: Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

Mon 28 SeptDistributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
The gist
Large language models save parts of their memory called key-value caches to speed up responses, but these caches can be too big for fast GPU memory. The authors designed TempoKV, which smarter manages when to move these caches into fast memory by predicting when they will be needed. This method saves memory resources and keeps the model running quickly even with less fast memory available. TempoKV was tested on real hardware and showed big improvements in speed and efficiency compared to previous methods.
Open → 2609.35065v1

Nested sequence parallelism speeds up training of long context large language models

NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism

Abstract: Long-context LLM training on long-tailed corpora faces a central communication--balance tradeoff. Such sequence-length heterogeneity makes any single sequence-parallelism (SP) degree a poor fit for the workload: a small degree leaves the few long sequences badly imbalanced, while a large degree forces the many short sequences that dominate the workload to pay excessive communication. Existing dynamic-SP systems mix SP degrees within a batch, but to run several groups at once they partition the GPUs into disjoint groups, which reintroduces imbalance across groups and forces costly micro-batch workarounds. We present NSP, a sequence-parallel training system that resolves this tradeoff by nesting differently sized SP groups on shared GPUs within a single training iteration. This lets long sequences use larger SP groups while keeping short sequences on smaller ones, so communication is incurred only where needed and load is balanced per GPU rather than per group. NSP realizes this idea with a tree-structured routing planner that assigns sequences under memory constraints and an executor that exploits the resulting hierarchy through inter-level phase streaming and tree-level recomputation. NSP supports common SP backends and requires no model changes. We evaluate NSP on Qwen3-MoE workloads with up to 384K-token contexts across multiple long-tail datasets on an internal production GPU cluster. Across these settings, NSP consistently improves end-to-end training throughput, outperforming Static SP by up to 1.48x and FlexSP by up to 1.16x.

Sat 19 SeptDistributed, Parallel, and Cluster Computing
The gist
Training large language models with very long and varied-length text sequences is slow because computers must balance work fairly while communicating enough data. The authors present NSP, a new method that groups sequences by length in a nested way on shared GPUs, so longer sequences get more resources and shorter ones get less, reducing wasted communication. This leads to faster overall training without changing the model itself. Tests on long-text datasets show NSP outperforms previous methods in training speed.
Open → 2609.22755v1