Hybrid memory setup boosts large language model session handling
Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Serving
Hardware Architecture
Summary
Running large language models (LLMs) for tasks where conversations pause and resume is tricky because computers have limited fast memory and slower storage that takes time to access. The authors show that LLM memory states have a small, frequently used 'hot' subset and a large, rarely used 'cold' subset. They suggest storing the hot data in fast but small memory and the cold data in slower but larger flash memory to speed up performance and save energy. Their approach lets a single GPU handle many more ongoing sessions with only a tiny delay when resuming paused conversations.
What this means in practice
- •For cloud ai service providers: Host many concurrent LLM conversations efficiently by splitting memory storage between fast GPU memory and high-capacity flash to reduce latency and energy use.
- •For gpu system architects: Design GPU memory hierarchies that optimize power use and capacity by storing frequently accessed LLM data in HBM and less frequent data in high bandwidth flash.
Authors
Jongjin Baek, Won Ji, Seungjae Yoo, Joo-Young Kim
Abstract
Large language model (LLM) serving is increasingly agentic, with multi-turn sessions that idle between actions yet must retain their full context. Limited GPU memory capacity forces inactive KV states to be evicted, so resuming a session incurs either costly recomputation or slow interconnect transfers. To address this, high bandwidth flash (HBF)-an on-package 3D-NAND memory offering orders-of-magnitude greater capacity than high bandwidth memory (HBM) at comparable read bandwidth-has emerged as a strong candidate. However, its high read energy and limited write endurance make it impractical to serve all KV traffic. Fortunately, our analysis shows that agentic KV states exhibit distinct access patterns: a small hot set is read for every decoding step, while a large cold pool is read only when a paused session resumes. Exploiting this, we place the hot set in HBM and the cold pool in HBF, forming a hot-cold KV hierarchy within the GPU memory tier. On agentic workloads with Qwen3-Coder-30B-A3B, our design delivers 14 ms time-between-tokens (TBT) and adds only $\approx$0.1 ms of resume latency on top of prefill, while hosting $24\times$ more concurrent sessions per GPU. By confining HBM to the hot set, our design also cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash-establishing HBF as a cold-tier complement to HBM rather than its replacement.