Unlocking the Working Memory of Large Language Models for Latent Reasoning
2026-05-28 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors noticed that current language models think by writing out each step one after another, which mixes up their thinking and speaking processes. They created a new method called Reasoning in Memory (RiM) that lets the model hold and work on information internally, like a human's working memory, without writing it all down. RiM uses fixed blocks of tokens to store thoughts and refines answers step-by-step in a more efficient way. Their tests show that RiM works as well or better than other methods while being faster and more memory-like.
large language modelsautoregressive generationworking memorylatent reasoningmemory blockstoken sequencesforward passcurriculum learningreasoning benchmarks
Authors
Lukas Aichberger, Sepp Hochreiter
Abstract
To improve the reasoning capabilities of large language models, test-time compute is typically scaled by generating intermediate tokens before the final answer. However, this couples reasoning to autoregressive generation and thereby conflates internal computation with external communication. In contrast, human cognition can use working memory to hold and manipulate information internally without the need to externalize intermediate thoughts. Drawing on this principle, we introduce Reasoning in Memory (RiM), a latent reasoning method that replaces the autoregressive generation of reasoning steps with memory blocks. These memory blocks are fixed sequences of special tokens that unlock the working-memory capacity of large language models. Since they are fixed rather than generated, they can be processed in a single forward pass, enabling compute-efficient latent reasoning. To operationalize these memory blocks, we employ a two-stage curriculum. First, we ground them by predicting explicit reasoning steps after each memory block. Second, we discard this step-level supervision and iteratively refine the final answer after each memory block. Our experiments on reasoning benchmarks show that, across language models of different families and sizes, RiM matches or exceeds existing latent reasoning methods while avoiding the autoregressive generation of thoughts. These results demonstrate that large language models can be trained to use working memory as an effective mechanism for latent reasoning.