Papers for

language model builders

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Transformers forget their prompt influence as generation grows

A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

Abstract: We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-α}$, where $α=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast ($α\approx0.55$--$0.7$), untrained ones slowly ($α\approx0.2$--$0.5$). We prove that this offset is $O\big(P(\log t)^{L-1}/t\big)$ for a prompt of length $P$ and depth $L$. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and \emph{relay} --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the \emph{conditional no-hair theorem}: given the text generated so far, the prompt's influence on the next token vanishes as $t$ grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards $1$ seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.

Fri 16 OctArtificial Intelligence

The gist

This paper studies how much a transformer language model remembers its initial prompt after it starts generating text. The authors find that the model's memory of the prompt fades over time, roughly following a mathematical decay where the prompt's effect gets weaker as more tokens are generated. Trained transformers forget faster than untrained ones, meaning their prompt influence disappears sooner. They also prove mathematically why this forgetting happens, showing that as more text is generated, the prompt's direct impact is diluted and only indirect signals pass along some information. Essentially, given enough generated text, the model's future predictions are shaped more by the generated text than the original prompt.

Open → 2610.24133v1

Feed-forward layers guide transformer tokens toward agreement dynamically

The Role of Feed-Forward Layers in Transformer Dynamics

Abstract: We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.

Mon 28 SeptMachine Learning

The gist

Transformers are computer programs that process language by passing information between tokens. This paper looks at how one part of these transformers—the feed-forward layer—helps guide tokens toward agreeing with one another regardless of how the attention part works. The researchers showed mathematically that the feed-forward layers can steer tokens to align closely, even pushing tokens into groups. They then tested their math with computer simulations and compared it to behavior seen in big language models.

Open → 2609.36230v1