High bandwidth flash improves large language model serving efficiency
Characterizing High Bandwidth Flash for LLM Serving
Machine LearningHardware ArchitectureDistributed, Parallel, and Cluster Computing
Summary
Serving large language models needs a lot of memory and fast data access. The authors explored using a type of fast flash memory to help store model data and make serving faster. They designed a system and scheduling method to use this memory efficiently without wearing it out too quickly. Their tests showed faster processing times and energy savings in many cases. This work helps computers run large language models more efficiently and sustainably.
What this means in practice
- •For cloud infrastructure teams: Improve latency and power use when serving large language models by adding high bandwidth flash memory with smart scheduling.
- •For data center operators: Extend storage lifetime and reduce energy costs in AI workloads by optimizing data placement and scheduling with high bandwidth flash.
Authors
Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami
Abstract
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.0% relative to HBM-only systems. Modeled energy savings reach 55.8%, although HBF increases energy consumption on some light workloads. Buffered cache-aware scheduling extends estimated HBF write lifetime from 4.77 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.