Latent reasoning states hold more thoughts with better memory balance
How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superposition
Artificial Intelligence
Summary
This work looks at how artificial intelligence can keep track of many ideas at once when solving complex problems. Instead of just remembering what they are thinking about now, models that remember the full history of their reasoning can use their limited memory space more efficiently. The authors find that keeping all past information in a balanced way helps the system make better decisions later. Surprisingly, focusing only on recent ideas can make memory less reliable and worse for reasoning.
What this means in practice
- •For machine learning engineers: Improve multi-step AI reasoning systems by using balanced cumulative memory representations inside latent states to keep more valid evidence accessible.
- •For natural language processing teams: Design better continuous reasoning models by uniformly weighting historical memories inside latent embeddings to avoid information loss and improve downstream prediction reliability.
Authors
Hongyu Gu, Chang Liu, Jingwen Fu
Abstract
Large language models solve hard problems through intermediate computations across multi-step reasoning. Traditional chain-of-thought encodes these computations as tokens. Recent continuous and recurrent methods instead move partial computations into fixed-dimensional latent states, where a single thought can superpose multiple alternatives. This raises a fundamental design question:what should continuous thoughts preserve as reasoning proceeds? An intuitive approach discards past computations and keeps only the current reasoning frontier. Storing more items seems to dilute states and waste limited representational capacity. We show this intuition can be incorrect. Under identical downstream computations, cumulative superposition retaining full reasoning history can require lower representational dimensions than frontier-only superposition holding only current alternatives. At fixed hidden width, this advantage allows latent reasoners to retain more valid evidence, distinguish more plausible downstream outcomes, and delay the point where compressed states turn unreliable. This counter-intuitive effect emerges because informative historical components coherently reinforce each other, while unrelated alternatives bring random interference. This perspective also answers a practical design question: how should models weight memories accumulated inside latent states when their future use is unknown? Across reusable weighted superpositions, prioritizing a small set of recent or salient items produces weakly-represented memories that bottleneck subsequent attention. Uniform cumulative weighting avoids this flaw, and we prove it is minimax-optimal for robust future reasoning. Our results turn superposition from an observed latent-space effect into a design principle: balanced cumulative memory lets a fixed representational budget support more reliable, reusable computations.