Adaptive KV Retention for LLM Agents at Human-Approval Timescales
2026-08-31 • Operating Systems
Operating Systems
AI summaryⓘ
The authors study how pausing and restarting large language model (LLM) requests, which can wait minutes or hours for human approval, affects GPU performance. They find that keeping paused data in GPU memory allows quick resuming but uses lots of GPU resources, while removing it saves resources but causes slow restarts. They create a smart system that balances these trade-offs by estimating costs in GPU time and deciding when to keep or discard paused data. Their method improves efficiency compared to existing systems when handling human-approval pauses.
LLM (Large Language Model)GPU servingsuspension and resumptionkey-value (KV) stategoodputretention policyhuman approval latencyvLLMMORIContinuum
Authors
Minseo Choi, Ananya Joshi
Abstract
Unlike the seconds-scale tool-call pauses targeted by prior agent-serving systems, agentic LLM requests can be suspended for minutes or hours while waiting for human approval. We study how suspension and resumption affect GPU serving performance and develop a retention policy that balances active-serving capacity against future recomputation under uncertain approval waits. The central tension is severe because retaining suspended KV preserves fast resume but can consume enough GPU capacity to reduce active-serving goodput by 41%, while evicting it avoids that residency cost at the expense of nearly $10\times$ higher resume latency when the request returns. We develop a tiered retention controller around GPU opportunity cost, which expresses the serving capacity consumed by preserving or reconstructing a suspended request's KV state in a common GPU-time cost. Within host memory, the controller selects between indefinite retention and load-indexed expiration using calibration wait samples, without requiring per-request wait prediction. On human-scale approval workloads, our controller improves active-request goodput by 23-51% over the vLLM baselines, 22-29% over MORI, and 41-52% over Continuum.