PATTON improves large language model serving using commodity processing in memory
PATTON: Enabling Commodity PIM for Production LLM Serving
Hardware Architecture
Summary
Serving large language models (LLMs) quickly and efficiently is challenging because it involves managing lots of data in memory. The authors present PATTON, a system that helps existing memory technologies work better for this by carefully organizing how data is stored and accessed. PATTON balances competing needs like fast calculations and efficient memory use, leading to almost twice the speed and much lower energy use compared to other methods. This innovation works without changing existing memory hardware and keeps performance similar to GPUs on cache hits.
What this means in practice
- •For cloud infrastructure teams: Enable faster and more energy-efficient serving of large language models using existing memory hardware.
- •For embedded system developers: Improve performance and reduce power consumption of memory-intensive AI tasks on commodity processing-in-memory devices.
Authors
Hangyeol Kim, Sanghyun Lee, Teokkyu Suh, Joo-Young Kim
Abstract
Processing-in-Memory (PIM) is promising for accelerating memory-bound decode attention, but attention acceleration alone is insufficient for production LLM serving, where engines dynamically allocate, populate, share, cache, and reclaim logical KV cache blocks. Supporting this lifecycle on commodity PIM requires efficient physical memory allocation, block-to-address mapping, and command generation. For the Value cache, these requirements create a fundamental conflict among GEMV efficiency, single-token write efficiency, and memory capacity: GEMV-optimized layouts scatter newly generated Value vectors across rows, making writes costly, while finer-grained memory sharing improves capacity utilization but fragments GEMV reductions. We present PATTON, a PIM runtime that integrates production LLM serving engines with commodity PIM. PATTON introduces hierarchical granule allocation: block-sized Key and Value granules map one-to-one to logical token blocks, fixing their physical placements and commands, while coarser granules group blocks for efficient GEMV execution and memory utilization. A Commit Zone stages partial Value blocks for efficient single-token writes before committing them to GEMV-optimized locations. PATTON tracks these placements to generate KV cache writes and QK-transpose/SV commands. Across attention execution and runtime-induced prefill recomputation, PATTON achieves an average 1.95x speedup and 4.83x higher energy efficiency over evaluated baselines, requires no PIM processing-unit modifications, and maintains a KV cache hit rate comparable to the native GPU KV cache in vLLM.