Janus speeds up large language model sessions using SSD storage
Efficient Agentic LLM Serving over SSD-based Sparse KV Storage
Distributed, Parallel, and Cluster Computing
Summary
Long conversations with large language models (LLMs) need a lot of memory to remember what was said before. The authors found a way to store and access these memories more efficiently by predicting which information the model will need next. They use fast solid-state drives (SSDs) smartly to avoid slowing down the model while it works. Their new system, Janus, helps LLMs respond faster without missing important details.
What this means in practice
- •For ai infrastructure engineers: Serve long multi-step LLM sessions faster by reducing memory bottlenecks using SSD-optimized key-value cache management.
- •For cloud service providers: Improve responsiveness of hosted LLM applications by integrating Janus to optimize disk-based memory usage during inference.$Commercial implications: Enables more cost-effective and scalable LLM serving infrastructure attractive to enterprise customers.
Authors
Wenhao He, Ping Zhang, Xiaohe Hu, Chutian Wang, Jinlong Hou, Yuan Cheng, Peng Sun, Fangcheng Fu
Abstract
Agentic sessions driven by Large language models (LLMs) often alternate between model inference and tool use, accumulating long histories across successive rounds. Serving these sessions efficiently requires reducing attention computation and retaining history key-value (KV) caches to avoid recomputation. Recently, frontier open-source LLMs adopt sparse attention to reduce computation by selecting only part of the history, while SSDs provide a cheaper alternative to CPU DRAM for storing KV caches. However, sparse KV selection depends on the ad hoc intermediate values during model inference, so it forces SSD reads to lie on the inference critical path. These reads are further slowed by fragmented accesses and read-write interference in SSDs. To address these challenges, we present Janus, an agentic serving framework for sparse attention LLMs with SSD-centric KV storage. Janus focuses on append prefill, which processes each round's newly added inputs and accounts for most history KV loading. To move SSD reads out of the critical path, Janus runs the model's own KV selection module on earlier intermediate values, predicting KV demand without additional training. The predicted reads overlap with model computation, and any prediction misses are fetched before attention executes to preserve model outputs. To improve SSD efficiency, Janus coalesces adjacent reads, packs scattered KV pages into sequential writes on the CPU, and limits background writes while reads are active. Across three models and three agentic traces, Janus outperforms existing works by up to 1.57-3.69 times (1.22-1.85 times on average) in terms of the time to first token latency, while maintaining decode efficiency.