High-bandwidth flash and smarter cache management boost AI recommendation speed and durability
Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy
Hardware Architecture
Summary
Generative recommendation systems use memory caches to quickly remember user preferences, but as these caches grow, managing them efficiently becomes harder. The authors explore using a type of fast flash memory that can hold more data than traditional memory and still read quickly. They found that the usual method for deciding what to keep in the cache causes too many writes, which wears out the flash memory fast. By using a new strategy that only saves information likely to be reused, the flash memory lasts much longer without slowing down the system. This approach makes recommendation systems faster and more sustainable over time.
generative recommendationkey-value cachehigh-bandwidth flashleast recently usedLRU-Kmemory bandwidthcache managementflash endurancecache admission policythroughput
Authors
Danni Peng, Kai Wu, Tianyu Zuo, Pengfei Xia, Hui Zang
Abstract
Generative recommendation (GR) systems increasingly leverage user-level KV cache reuse to avoid recomputing long user histories. However, the growing KV cache capacity and bandwidth requirements introduce new challenges for memory system. High-Bandwidth Flash (HBF) provides a promising solution by offering substantially higher capacity than HBM while approaching HBM-class read bandwidth, enabling larger scale KV cache retention and improved serving throughput. Yet conventional Least-Recently-Used (LRU) KV cache management tightly couples KV cache writes with cache misses, generating excessive write traffic that rapidly exhausts flash endurance. In this work, we evaluate a write-aware KV cache policy based on admission-controlled LRU-K for HBF-based GR serving. By filtering low-reuse users before cache admission, LRU-K decouples KV cache writes from misses and significantly reduces unnecessary writes. We develop an analytical model to characterize GR serving performance, KV cache write traffic, and HBF lifetime, and evaluate performance across diverse memory systems and GR workloads. Our results show that HBF-based systems achieve 3.8 to 4.7 times higher throughput than HBM-only systems. Moreover, LRU-K extends HBF lifetime from about one year under conventional LRU to over six years with a moderate K=10, while maintaining comparable or even slightly improved throughput. These results highlight the importance of write aware KV cache policy for sustainable HBF-based GR serving.