Language models recall limits grow predictably with model size
In-Context Binding Capacity in Language Models
Machine Learning
Summary
Language models can remember certain pairs of information but only up to a point before mixing them up. The authors measured how many pairs different-sized models can keep straight, finding the capacity grows less than proportionally with size. They also found that how a model is trained can affect memory capacity, but the effect disappears when considering model size alone. Their work helps understand the memory limits of these AI systems without linking it to how well they follow instructions or update knowledge.
What this means in practice
- •For language model developers: Assess and predict how many distinct associations a model can reliably recall as its size changes to guide architecture design.
- •For ai system engineers: Evaluate memory constraints in designing systems that rely on language models for tracking multiple pieces of information simultaneously.
Authors
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
Abstract
How many assignments can a language model recall before it loses track of which value belongs to which entity? We measure this limit using continuous recall curves for 12 models at or below 3B parameters and a threshold sweep over 30 open models up to 12B. On the continuous curves, the load at which recall falls halfway to chance follows $K_{50}=cN^α$, with $α=0.820$ and $R^2=0.73$. The broader sweep shows an eightfold range associated with pretraining recipe, although the continuous curves show no detectable recipe effect after controlling for scale, with few modern models in the fit. We derive why interference can lower measured capacity by reducing single-binding recall even when the load-dependent recall profile is unchanged. Direct task training also exceeds the extrapolated zero-shot law, but different measurement criteria prevent interpreting that comparison as a capacity gain. Its formation times follow a power-law form in two independent codebases, conditional on runs that succeed. Together, these results characterize capacity at the model's query interface. Bounds on joint recall and a decomposition of policy errors connect this measurement to working memory and instruction following, without treating recall as a measure of alignment. The controlled task also provides a baseline for testing whether binding limits constrain world-state tracking; the present experiments do not measure state updates or downstream transfer.