Architecture speeds large language model serving with smart quantization
A Shape-Adaptive Architecture with Disaggregated Quantization for Efficient LLM Serving
Hardware Architecture
Summary
Large language models are powerful but slow and complex to run efficiently because they work in two phases that use computing differently. The authors designed a new computer architecture called DynaCore that adapts its shape and precision based on these phases to use resources better. DynaCore also picks the best setup for each batch of work to keep things fast. Testing it on real workloads showed much faster responses than existing approaches.
What this means in practice
- •For data center operators: Reduce latency and improve throughput of AI inference servers running large language models by using DynaCore’s adaptive hardware and quantization.
- •For hardware designers: Design more efficient accelerators for LLM inferencing by implementing shape-adaptive and disaggregated quantization features inspired by DynaCore.
Authors
Cong Guo, Chiyue Wei, Bowen Duan, Haoxuan Shan, Benjamin F. Morris, Yintao He, Hai "Helen" Li, Yiran Chen
Abstract
Large language models (LLMs) have become the backbone of modern AI applications, but pose significant challenges for efficient inference. Their autoregressive generation divides execution into two phases: prefill, dominated by large GEMMs, and decoding, dominated by small GEMVs. Modern serving systems further introduce complexity through continuous batching and prefill-decoding disaggregation, leading to dynamic workloads and phase separation. However, existing accelerators remain poorly aligned with these system-level behaviors, resulting in inefficiencies in LLM serving. In this work, we present DynaCore, a unified architecture for efficient LLM serving via system-architecture co-design. We observe that the compute tile a systolic array executes, its Minimum Efficient Unit (MEU), spans all three GEMM dimensions. DynaCore reshapes the MEU along all three: spatially it trades array width against height asymmetrically, raising weight delivery while leaving the input path untouched, and temporally Split-K maps the reduction onto the array, folding partial sums through the interconnect the array already has. To exploit phase separation, we further propose disaggregated quantization, applying dual-side quantization to prefill and weight-only quantization to decoding, with an inner-product mixed-precision datapath that keeps output width invariant to precision. A runtime scheduling framework then selects an MEU per batch. Evaluation with real-world serving traces shows that DynaCore substantially reduces service-level latency over quantization and reconfigurable accelerators, improving TTFT by 3.50x and 2.97x and TPOT by 36.55x and 8.02x, respectively.