Most transformer decisions rely on a small set of key components
Through the Looking Glass: Directly Reading and Writing Transformers
Computation and LanguageMachine Learning
Summary
Transformers, a type of AI model used for language tasks, make predictions based on many internal parts called components. This work found that while thousands of parts contribute to each prediction, only a small handful are really needed to produce or destroy it. The authors showed that just a few components—sometimes as few as two—can explain most of what drives a prediction. They also demonstrated how specific components can be read and modified directly to change model behavior without retraining. This sheds light on how transformer models represent information and make decisions.
What this means in practice
- •For machine learning engineers: Edit transformer model behavior directly by modifying a small number of identified components to customize predictions without retraining.
- •For natural language processing teams: Analyze prediction contributions in transformer models to improve interpretability of language model decisions and debugging.
Authors
Mark Oskin
Abstract
How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the predicted token is a median of seven times the mass carrying it. Divide by the net and the count is dozens: on the baseline, 53 components carry ninety percent of a prediction, 13 it cannot survive losing, and 8 suffice to produce it alone. Across twelve models trained elsewhere, 124M to 7B parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size. Three quarters of a layer's update is a fixed linear map of the state it received. Everything is read from the model's own parameters and activations, with nothing trained or fitted, and it names a component on both sides: what it writes, from the predictions it drives, reaching close to half of every model; what it reads, from its weights in the frame of its own layer, at 58.9 percent above chance over its eight strongest inputs. Sorting the remainder by upstream source yields grammatical categories the embedding cannot see. A name can be acted on. An association the model does not hold installs into one spare unit, key and value read from the weights, for a quarter of a percent of held-out loss, a fortieth of what a rank-one update costs. An installed attention head and a unit two layers above it make an edit fire only where a token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream, 86 percent of the effect passing through it. An order-preserving activation puts a unit's inputs at the instrument's ceiling, at the price of a two-part install.