Summary
This paper studies how much a transformer language model remembers its initial prompt after it starts generating text. The authors find that the model's memory of the prompt fades over time, roughly following a mathematical decay where the prompt's effect gets weaker as more tokens are generated. Trained transformers forget faster than untrained ones, meaning their prompt influence disappears sooner. They also prove mathematically why this forgetting happens, showing that as more text is generated, the prompt's direct impact is diluted and only indirect signals pass along some information. Essentially, given enough generated text, the model's future predictions are shaped more by the generated text than the original prompt.
What this means in practice
- •For language model builders: Design prompt engineering techniques that account for the limited memory of prompts over long generated sequences.
- •For machine translation developers: Optimize translation models by understanding how prompt influence fades and ensuring relevant context is preserved in generated text.
A theory result. No direct application yet.
Abstract
We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-α}$, where $α=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast ($α\approx0.55$--$0.7$), untrained ones slowly ($α\approx0.2$--$0.5$). We prove that this offset is $O\big(P(\log t)^{L-1}/t\big)$ for a prompt of length $P$ and depth $L$. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and \emph{relay} --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the \emph{conditional no-hair theorem}: given the text generated so far, the prompt's influence on the next token vanishes as $t$ grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards $1$ seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.