Heterogeneous 3d chip speeds up large language model inference fivefold
Tri-Stack-PIM: A Heterogeneous Multi-Die 3D-Integrated Processing-in-Memory Architecture for Efficient LLM Inference
Hardware Architecture
Summary
Running large language models (LLMs) can be very slow because moving data between memory and processors is a bottleneck. The authors propose a new chip design called Tri-Stack-PIM that stacks three different layers of memory and processors to work together more efficiently. This design dedicates each layer to a specific task, improving speed and saving energy. They also created a smart scheduler to better coordinate tasks across these layers, recovering most lost cycles from simpler methods. Their tests show significant speed gains compared to current graphics processors and other specialized designs.
What this means in practice
- •For ai hardware engineers: Design chips that speed up LLM inference by integrating memory and compute across multiple stacked dies to reduce data movement delays.
- •For cloud service operators: Deploy advanced multi-die memory-compute chips to accelerate LLM workloads, lowering energy use and improving throughput for AI services.
Authors
Shuao Jia, Zichao Ling, Chen Bai, Kang Zhao, Jianwang Zhai
Abstract
Large language model (LLM) inference during autoregressive decoding is fundamentally memory-bandwidth-bound. Existing three-dimensional (3D) stacked processing-in-memory (PIM) architectures alleviate the off-chip bandwidth wall, yet remain constrained by several limitations: physical layer (PHY), controllers, and direct memory access (DMA) engines compete with arithmetic units for advanced-node base-die area; hybrid-bonding keep-out zones fragment the base-die floorplan and impede physical design; compute tiles built in a dynamic random access memory (DRAM) process are restricted to low logic density, capping achievable throughput; and nonlinear operations offloaded to the host reintroduce a communication bottleneck. We present Tri-Stack-PIM, a heterogeneous three-die architecture that resolves these tensions by dedicating a top DRAM die to storage, a middle buffer die on a mature process node to PHYs, DMA engines, and near-memory compute units, and a bottom base die on an advanced node exclusively to single-instruction-multiple-data (SIMD) multiply-accumulate (MAC) arrays. We further introduce a Unified Buffer-Aware Ready-based (UBAR) hardware out-of-order micro-operation scheduler that enforces cross-die dependency correctness at runtime while maximizing issue-slot utilization through a multi-term priority function, recovering up to 90% of the cycles that an in-order policy would forfeit. A joint architecture-dataflow co-exploration identifies Pareto-optimal configurations that reduce the energy-delay product (EDP) by 33-66x over naive parameter selection. Evaluated across multi-head, grouped-query, and multi-query attention (MHA, GQA, MQA) mechanisms with batch sizes 1-32, Tri-Stack-PIM achieves geometric-mean speedups of 4.95x, 2.57x, and 1.68x across all three attention mechanisms and batch sizes over a graphics processing unit (GPU) baseline, AttAcc, and H2-LLM, respectively.