Flexible memory system speeds up large language model processing
HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution
Hardware Architecture
Summary
Large language models need a lot of memory to run well, but current memory systems struggle with balancing the demands of reading, writing, and managing temporary data. The authors created a new memory system called HBFlex that uses high-bandwidth flash storage more efficiently by carefully organizing data placement, combining updates, and managing cleanup tasks. Their system improves the speed and efficiency of memory operations when running these big models. They tested HBFlex with simulations and found it was faster than previous flash-based memory systems.
What this means in practice
- •For cloud service providers: Improve large language model serving speed and capacity by using efficient flash-based memory management with HBFlex.$Commercial implications: Enables providers to offer faster LLM inference services using cost-effective flash storage with optimized memory handling.
- •For data center operators: Reduce memory bottlenecks and improve overall throughput of LLM-based applications by adopting HBFlex's memory system design.
Authors
Shuzhang Zhong, Weikai Xu, Yifan Zhou, Tongbin Zhao, Tenghao Zhao, Yifei Kang, Cunyin Chang, Shu Li, Guangyu Sun, Meng Li
Abstract
Large language models (LLMs) require increasing memory capacity to accommodate growing model weights and KV caches. High-Bandwidth Flash (HBF) offers high memory density and aggregate read bandwidth through massive plane-level parallelism, making it an attractive option for LLM serving. However, serving LLMs entirely from HBF introduces three challenges: fine-grained KV reads create placement and access imbalance, incremental writes interfere with foreground reads, and mixed KV lifetimes amplify garbage collection. Hybrid HBM/HBF designs retain HBM to support dynamic KV management, but this allocation reduces the HBF resources available under a fixed packaging budget, limiting aggregate HBF bandwidth. We present HBFlex, a full-HBF memory system with coordinated optimizations for KV reads, writes, and reclamation. HBFlex balances KV placement and attention accesses to improve plane utilization. It aggregates incremental updates and schedules writeback within sufficiently long compute windows to reduce write--read interference. It also combines lifetime-guided block packing with deferred reclamation to reduce valid-page migration. We evaluate HBFlex through trace-driven simulation across different configurations. HBFlex achieves average throughput speedups of up to 1.58$\times$ over FlashAccel and 3.30$\times$ over H3, benefiting from higher HBF bandwidth and more efficient management of dynamic KV-cache reads, writes, and erases.