Gpu and Npu share language model work for faster inference
HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference
Hardware ArchitectureDistributed, Parallel, and Cluster Computing
Summary
Large language models have two parts when they work: one part needs a lot of computing power (prefill), and the other mostly needs memory (decode). GPUs are great at the first part but waste power during the second, while NPUs are better at the second part. The authors created HAWK, a method that helps move data quickly and efficiently from GPUs to NPUs, even though these devices handle data differently. This makes the combined system almost as fast as just GPUs alone but uses less power and cost.
What this means in practice
- •For cloud ai service providers: Use heterogeneous GPU-NPU systems to run large language models more efficiently by splitting computation phases and transferring data with HAWK.$Commercial implications: Enables service providers to reduce operational costs and power consumption while maintaining low-latency AI model inference.
- •For hardware accelerator designers: Design systems that optimize data transfer and phase-specific workloads across GPUs and NPUs to improve LLM inference performance and efficiency.
Authors
Junhyeok Park, Jungeun Yoon, Kyungmin Kim, Hunsoon Lee, Shinyoung Ahn, Kangho Kim
Abstract
Large language model (LLM) inference exhibits distinct computational characteristics across its two phases: prefill is compute-intensive, whereas decode is more memory-bound. As a result, AI accelerators achieve varying levels of efficiency across the two phases. GPUs are well suited to prefill, but are often underutilized during decode. By contrast, NPUs can utilize their compute resources more effectively during decode. Motivated by this observation, we explore a heterogeneous architecture that integrates GPUs and NPUs, mapping prefill to GPUs and decode to NPUs to achieve high performance and efficiency. However, realizing such heterogeneous integration is challenging because LLM phase disaggregation requires KV cache transfer from prefill to decode (that is, from GPUs to NPUs), which is complicated by differences in inter-device communication mechanisms and KV cache tensor layouts. To address this, we propose HAWK, a transfer mechanism that enables rapid KV cache transfer across heterogeneous devices. HAWK establishes a remote direct memory access (RDMA) path for direct cross-device transfers while handling inter-device tensor layout discrepancies as part of the transfer process. In particular, HAWK allocates tensor-layout-aware RDMA buffers on the NPU side to match the target tensor layout. Leveraging these buffers, HAWK orchestrates data transfer from a conventional contiguous GPU RDMA buffer to the corresponding NPU buffers, allowing transferred tensors to be consumed directly without additional data rearrangement while maintaining transfer efficiency. We implement and evaluate HAWK on a system built with commercially available GPUs and NPUs, demonstrating performance within 7.88% of a GPU-only system under the same SLO constraints while achieving up to 1.26$\times$ higher power efficiency and 1.72$\times$ higher cost efficiency.