Papers for

gpu system engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Gpu and memory work together to speed up language model answering

AMEND: Audited Margins Enable Nonblocking Drops in GPU-PIM LLM Decoding

Abstract: Autoregressive large language model (LLM) decoding re-reads a growing key-value (KV) cache at every step, so long-context attention is bound by graphics processing unit (GPU) memory bandwidth. Block-sparse attention skips low-contribution KV blocks, but a selector that decides after the current query-key (QK) product, such as max-relative block thresholding (BLASST), still reads every K block, and a processing-in-memory (PIM) filter that decides from the current query places a serial PIM stage on the critical path. We present AMEND, a GPU-PIM attention design that removes both dependencies. AMEND predicts each block's BLASST verdict from margins audited at earlier steps, so the GPU fetches only predicted survivors while near-bank PIM units in high-bandwidth memory (HBM) concurrently score the omitted complement. A stack-level controller merges both observations, updates the predictor, and eagerly generates the next step's mask, so every predicted drop is re-observed without blocking the current token. Operating points are selected offline by constrained Bayesian optimization under a false-drop budget. Across LongBench and RULER runs, AMEND preserves near-baseline task quality; in simulation at batch size 8, it achieves $1.40$-$3.63\times$ end-to-end decode speedup and 28-66% lower dynamic decode energy than dense attention across 8K-64K contexts.

Wed 9 SeptHardware Architecture
The gist
When large language models generate text, they must repeatedly read a lot of stored information, which slows things down. The authors designed a way for GPUs and special memory units to predict which parts can be skipped and which must be checked, working in parallel without waiting. This method speeds up text generation by up to three times and uses less energy, while keeping the quality of the generated text close to normal. They tested their approach on tasks that involve very long text to ensure it works well.
Open 2609.09823v1