Summary
Large language models like transformers can learn from examples given in their prompt without changing their weights—a way called in-context learning. This paper shows that transformers can actually mimic complex data generation processes, like sampling from probability distributions, using only their internal computations and the examples given. The authors reveal that transformer attention works like weighting and averaging sample points while the feedforward part updates these estimates iteratively. They also observe patterns in the model's internal states that match known energy-based sampling methods, linking the theory with what happens inside real pretrained language models.
What this means in practice
- •For machine learning engineers: Design more sample-efficient generative models by harnessing transformer in-context learning as implicit samplers without retraining.
- •For natural language processing developers: Enhance prompt engineering by understanding how transformers internally generate topic-consistent samples from example prompts.
A theory result. No direct application yet.
Authors
Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh, Hadi Daneshmand
Abstract
A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.