Simulator models flash memory impact in large language model computing
HBFSim: A Simulator for Exploring HBF in Memory Hierarchies for LLMs
Hardware ArchitectureEmerging Technologies
Summary
Large language models need lots of memory to work well, but different types of memory have trade-offs between speed, size, cost, and lifespan. The paper’s authors created HBFSim, a simulator that helps understand how a new kind of memory called High Bandwidth Flash (HBF) behaves when combined with traditional high-speed memory. Their tool predicts how well HBF can speed up processing and how long it lasts under real workloads. They found that HBF is much faster than standard memory at handling data that doesn’t fit in the fastest memory and lasts years under typical use.
What this means in practice
- •For hardware architects: Design memory systems integrating high bandwidth flash with HBM to optimize large language model performance and reliability.
- •For llm serving engineers: Predict system latency and flash memory lifetime when deploying large language models using a combination of HBM and flash memory tiers.
Authors
Jinheng Li, Guosheng Wang, Hongchao Du, Shangyu Wu, Qiao Li, Chun Jason Xue
Abstract
High Bandwidth Flash (HBF) has recently emerged as a promising technology that provides terabyte-scale accelerator-local capacity at bandwidths approaching HBM, with a potential cost-per-bit advantage over HBM. However, its NAND foundation brings long latency, out-of-place updates, and finite write endurance directly into the accelerator memory hierarchy, making its effects on performance and lifetime difficult to predict. To study this emerging design space, we present HBFSim, an LLM-aware event-driven simulator for heterogeneous HBM+HBF memory systems that preserves application-level memory semantics while modeling flash-internal behavior. HBFSim models HBM, HBF with host-managed flash translation, and conventional capacity tiers in one simulation driven by a calibrated serving frontend. Using HBFSim, we study decoding of Llama-3.1-70B and Qwen3-235B-A22B under three workloads that differ in their input and output lengths. HBF serves data that does not fit in HBM 11.0--19.0$\times$ faster than host DRAM; HBM is better used to hold the running sequences' KV cache than as a cache for HBF; and at a published peak load of Llama-3.1-70B, HBF lasts 2.0 to 6.9 years, in proportion to the stacks that share the KV writes. The artifact is available at https://github.com/FlashLabAE/HBFSim and will track public HBF device data as it becomes available.