High-bandwidth flash improves generative recommendation serving speed and lifespan
Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy
Hardware Architecture
Summary
Generative recommendation systems remember past user behavior to make better suggestions, but storing all this data needs a lot of fast memory, which can be costly and short-lived. The authors show that using a special type of flash memory with a smarter way to store data can keep the system running much faster and for many more years. Their new method avoids unnecessary writing to the memory, keeping the memory healthy longer while still making recommendations quickly.
What this means in practice
- •For cloud infrastructure teams: Use high-bandwidth flash with write-aware cache policies to scale recommendation systems with lower memory costs and longer hardware lifespan.
- •For data center operators: Extend the lifetime of flash memory in serving hardware by filtering low-reuse data to reduce write wear, minimizing hardware replacement needs.
Authors
Danni Peng, Kai Wu, Tianyu Zuo, Pengfei Xia, Hui Zang
Abstract
Generative recommendation (GR) systems increasingly leverage user-level KV cache reuse to avoid recomputing long user histories. However, the growing KV cache capacity and bandwidth requirements introduce new challenges for memory system. High-Bandwidth Flash (HBF) provides a promising solution by offering substantially higher capacity than HBM while approaching HBM-class read bandwidth, enabling larger scale KV cache retention and improved serving throughput. Yet conventional Least-Recently-Used (LRU) KV cache management tightly couples KV cache writes with cache misses, generating excessive write traffic that rapidly exhausts flash endurance. In this work, we evaluate a write-aware KV cache policy based on admission-controlled LRU-K for HBF-based GR serving. By filtering low-reuse users before cache admission, LRU-K decouples KV cache writes from misses and significantly reduces unnecessary writes. We develop an analytical model to characterize GR serving performance, KV cache write traffic, and HBF lifetime, and evaluate performance across diverse memory systems and GR workloads. Our results show that HBF-based systems achieve 3.8 to 4.7 times higher throughput than HBM-only systems. Moreover, LRU-K extends HBF lifetime from about one year under conventional LRU to over six years with a moderate K=10, while maintaining comparable or even slightly improved throughput. These results highlight the importance of write aware KV cache policy for sustainable HBF-based GR serving.