RedKnot-MLA improves memory reuse for long-context AI model serving
RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving
Artificial Intelligence
Summary
Handling very long text inputs for AI models can be slow and use a lot of memory. The authors present RedKnot-MLA, a method that cleverly splits the work between offline and online processing to reuse memory efficiently. This speeds up how fast model responses start and saves computation, especially for very long documents. Their system works by preparing document parts ahead of time and adjusting them when queries arrive to avoid full recomputation. It improves accuracy slightly overall but has tradeoffs on some specific datasets.
What this means in practice
- •For machine learning engineers: Speed up AI models handling very long documents by splitting precomputation and online reuse for greater efficiency and faster response times.
- •For cloud service operators: Reduce computational costs and improve throughput when serving large-scale long-context transformer models by applying memory reuse techniques across query sessions.
Authors
Yang Liu, Zhaokai Luo, Huayi Jin, Ruozhou He, Chenchen Hong, Mingxiao Ma, Biao Zhang, Zhiyong Wang, Boyu Wang, Guanjie Chen, Yifei Liu, Tao Xie, Junhao Hu
Abstract
Multi-head latent attention (MLA) exposes many logical query heads through one packed latent KV stream. This representation is memory efficient, but it removes the physical per-head cache boundary assumed by conventional head-wise reuse. We present our system, a DeepSeek-V4 realization of RedKnot's head-aware reuse principle. Each immutable document is processed offline at canonical position zero; certified Local-head contributions are retained as MLA-Off. At serving time, query-side RoPE relocation restores the document's request position, a small Global-head set and protected Local token rows are recomputed as MLA-Online, and the two paths are merged before a single shared output projection. The packed MLA latent is never split. DeepSeek-V4-Flash uses 37 reusable layers and a 56/8 Local/Global partition, giving a 75.29% analytic logical head-row ceiling; the Pro-0813 profile uses 55 layers and 112/16 heads, giving 78.89%. Frozen Flash operating points show hot-artifact TTFT speedups of 2.02-3.84x. At 256K, the archived three-dataset study reports an aggregate F1 change of +3.24 percentage points, an EM change of +4.16 points, and a 78.7-79.5% analytic major-operator arithmetic saving, while one dataset decreases by 2.81 F1 points. A separate author-reported 256K hot-artifact QPS measurement is approximately 2.0x; because its raw concurrency trace is not included in this bundle, we mark it as preliminary rather than archived evidence. We describe the factorization, position repair, token-row closure, sparse-MoE support, TP8 integration, and the measurement boundaries needed to interpret these results.