HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators
2026-06-29 • Hardware Architecture
Hardware ArchitectureDistributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors study how to run large language models (LLMs) efficiently by splitting the work onto different types of hardware: one kind with cheaper, slower memory for the first step (prefill) and another with faster, more expensive memory for the second step (decode). They create a system called HMA-Serve that lets these different hardware units work well together, even if they come from different vendors with different software and data formats. Their method uses smart ways to compress data and overlap tasks to speed up results without losing quality. This approach improves performance and cost-effectiveness significantly compared to traditional single-hardware solutions.
LLM inferenceprefill phasedecode phaseHBMGDDRquantizationKV cachetime-to-first-tokenmemory-heterogeneous acceleratorsgoodput
Authors
Zhixiang Wei, Yun Wang, James Yen, Mingyuan Xia, Zhengwei Qi
Abstract
LLM inference comprises a compute-bound prefill phase and a memory-bound decode phase, and recent systems disaggregate them onto separate hardware. Yet today's datacenter GPUs rely on costly HBM whose bandwidth sits almost entirely idle during prefill. LLM serving across memory-heterogeneous accelerators (MemHA) pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, promising lower cost without sacrificing performance. Pushed to its most economical form, MemHA serving is inherently cross-vendor, since the best-suited chip for each phase may come from a different vendor. This breaks two assumptions that single-vendor disaggregation takes for granted -- a KV format both ends consume natively, and a shared software stack. We present \textbf{HMA-Serve}, a MemHA-centric disaggregated serving system pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode efficiently. HMA-Serve achieves this through (1) phase-wise quantization, applying vendor-native low precision for high-throughput prefill while keeping decode in high-precision BF16, (2) a compute-transfer pipeline that overlaps each layer's KV cache transfer with later-layer prefill to reduce time-to-first-token (TTFT), and (3) deferred dequantization, shipping raw quantized bytes and reconstructing them lazily on the decode GPU to reduce network bandwidth and HBM usage. Across four Qwen3 models (4B--32B) and three production traces, HMA-Serve delivers up to $3.2\times$ higher goodput than state-of-the-art memory-homogeneous methods and $4.8\times$ higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks.