Large language model remembers 50 million tokens faster and cheaper
Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute
Computation and LanguageArtificial IntelligenceDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Large language models usually have to re-read everything in their memory every time they answer, which takes time and energy. The authors tested a special memory system that saves parts of the model’s understanding on a fast encrypted disk, so it can quickly reload them without re-doing all the work. This system worked well on 50 million tokens and was much faster and less energy-hungry than the usual way. The models were also good at answering questions about information from very long ago, showing the system can support very long memories.
What this means in practice
- •For ai infrastructure engineers: Reduce inference time and GPU energy by reusing saved model states across long text inputs instead of recomputing them.
- •For enterprise chatbot developers: Maintain and recall facts planted millions of tokens earlier efficiently by loading stored KV states to support longer contextual memory.
Authors
Sietse Schelpe
Abstract
A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models. Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer. The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.