Papers for

data center architects

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Trillion-parameter AI models hosted efficiently with flash storage

Trillion-Parameter MoE in a Box: Decoupling Memory Provisioning with High-Bandwidth Flash

Abstract: An MoE appliance for trillion-parameter models at low concurrency must host terabytes of weights on one node and serve prefill and decode with fixed resources. Combining operator analysis of two trillion-parameter MoE models, a measured expert routing trace, and agentic serving traces over multiple turns, we explore a design space spanning High-Bandwidth Flash (HBF) and DRAM configurations, bandwidth exposure, and near-data compute. We find that state bandwidth and HBF transport form two largely orthogonal knees and address two provisioning questions. Q1: Once weights move to HBF, what bandwidth-to-capacity ratio does DRAM require? With a 256-GB floor for the state tier, both models meet a $1.10\times$ completion time target at ratios of only $1.4$--$4.0~\mathrm{s}^{-1}$, roughly an order of magnitude below HBM3e's $33.3~\mathrm{s}^{-1}$. Q2: As HBF internal bandwidth scales with capacity, must bandwidth to the host scale proportionally? Six HBF packages expose 384~GB/s per package to the host, 2.30~TB/s aggregate and 62.5\% below the 6.14~TB/s full exposure reference, while more packages reduce required bandwidth per package at the same target.

Mon 14 SeptHardware Architecture
The gist
Very large AI models with trillions of parameters need enormous memory to work well. The authors studied how high-speed flash storage can be used together with smaller amounts of fast memory (DRAM) to host these massive models on a single machine. They found ways to balance memory size and data transfer speeds so the models perform almost as fast as if they used expensive high-bandwidth memory. This could make deploying huge AI models more practical and cost-effective.
Open → 2609.15636v1