Hybrid GPU and memory design boosts language model efficiency
A Hybrid GPU and Processing-Near-Memory Architecture with Shared DRAM for Efficient Prefill-Decode Disaggregation
Hardware Architecture
Summary
Running big language AI models needs lots of computing power and memory. The authors found that when two parts of model processing happen together, they slow each other down, and doing them separately wastes resources. They created a new system called DuoPD that combines special processing units near memory with a GPU to run these parts at the same time more smoothly. This approach uses both computing and memory better, making language model tasks run over twice as fast as before.
What this means in practice
- •For cloud infrastructure teams: Improve server efficiency for AI language model inference serving by enabling concurrent compute and memory-bound operations without resource waste.
- •For hardware designers: Design hybrid GPU-memory integrated chips that run large model inference more efficiently by embedding processing units near memory.
Authors
Sanghyeok Han, Inseong Choi, Byungkuk Yoon, Gyeonghwan Park, Jae-Joon Kim
Abstract
The growing adoption of large language models (LLMs) has significantly increased the compute and memory demands of inference serving. A key challenge arises from the mismatch between compute-bound prefill and memory-bound decoding: when executed together, the two phases interfere, while separating them via prefill-decode (PD) disaggregation leads to substantial resource underutilization. Specifically, prefill leaves memory bandwidth underutilized, whereas decoding underutilizes compute resources. Prior approaches, such as GPU Streaming Multiprocessor (SM) partitioning, attempt to improve utilization by spatially dividing GPU resources. However, these approaches remain fundamentally limited, as memory-bound decoding still requires a large number of SMs to saturate bandwidth, reducing the compute resources available for prefill and resulting in an inherent trade-off. To address this limitation, we propose DuoPD, a hybrid PD-disaggregation architecture that integrates lightweight Processing-Near-Memory (PNM) units into the HBM logic die. Unlike prior PIM/PNM designs that rely on costly internal bandwidth scaling and offload only partial decoding operations, DuoPD enables full decoding execution on PNM by leveraging bandwidth sharing with the GPU over a shared HBM interface. This allows compute-bound prefill and memory-bound decoding to execute concurrently with minimal interference, improving both compute and memory utilization. To realize this design, we develop a compact PNM microarchitecture, a TSV-level bandwidth sharing mechanism, and a hierarchical data mapping scheme tailored for hybrid GPU-PNM execution. Evaluations on server-scale LLMs demonstrate that DuoPD achieves up to 2.23x higher end-to-end throughput than conventional PD-disaggregation.