Trillion-parameter AI models hosted efficiently with flash storage
Trillion-Parameter MoE in a Box: Decoupling Memory Provisioning with High-Bandwidth Flash
Hardware Architecture
Summary
Very large AI models with trillions of parameters need enormous memory to work well. The authors studied how high-speed flash storage can be used together with smaller amounts of fast memory (DRAM) to host these massive models on a single machine. They found ways to balance memory size and data transfer speeds so the models perform almost as fast as if they used expensive high-bandwidth memory. This could make deploying huge AI models more practical and cost-effective.
What this means in practice
- •For data center architects: Design AI servers that combine flash and DRAM to host and serve trillion-parameter models efficiently with fixed hardware resources.
- •For enterprise cloud providers: Provide scalable inference for large AI models by optimizing memory provisioning to reduce costly high-bandwidth memory requirements.$Commercial implications: Enables cost-effective hosting of massive AI models for cloud AI inference services serving multiple concurrent users.
Authors
Pengfei Xia, Tuo Hao, Shengwei Li, Jinjing Chen, Shiru Wei, Wenjun Zou, Rui Zhang, Hui Zang
Abstract
An MoE appliance for trillion-parameter models at low concurrency must host terabytes of weights on one node and serve prefill and decode with fixed resources. Combining operator analysis of two trillion-parameter MoE models, a measured expert routing trace, and agentic serving traces over multiple turns, we explore a design space spanning High-Bandwidth Flash (HBF) and DRAM configurations, bandwidth exposure, and near-data compute. We find that state bandwidth and HBF transport form two largely orthogonal knees and address two provisioning questions. Q1: Once weights move to HBF, what bandwidth-to-capacity ratio does DRAM require? With a 256-GB floor for the state tier, both models meet a $1.10\times$ completion time target at ratios of only $1.4$--$4.0~\mathrm{s}^{-1}$, roughly an order of magnitude below HBM3e's $33.3~\mathrm{s}^{-1}$. Q2: As HBF internal bandwidth scales with capacity, must bandwidth to the host scale proportionally? Six HBF packages expose 384~GB/s per package to the host, 2.30~TB/s aggregate and 62.5\% below the 6.14~TB/s full exposure reference, while more packages reduce required bandwidth per package at the same target.