What It Costs to Compose, Rebuild, and Correct Precomputed Memory
2026-08-31 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors study how language models use precomputed memory—saved information from reading materials ahead of time—to answer questions faster without rereading everything each time. They find that this approach works well only if the memory is rebuilt regularly as new information comes in, because older precomputed memory can become outdated or incorrect. They suggest ways to update memory efficiently, like rebuilding trained compressed caches or adding specific updates alongside the memory, which help keep answers accurate between full rebuilds. Their work highlights the trade-offs between speed, accuracy, and updating frequency when using precomputed memory in real applications.
language modelprecomputed memorykey-value cachecache compressionmodel rebuildingmemory updatecontextLlama-3.1-8B-Instructreal-time informationmodel inference
Authors
Asa Shepard
Abstract
Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.