Reading transformer differences reveals model-specific token patterns
Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses
Computation and Language
Summary
Transformer models can be hard to understand because their internal states often reflect common predictions rather than specific details. The authors show that by comparing the differences between similar inputs inside the model, they can highlight what truly distinguishes those inputs. This approach helps reveal how models process complex ideas like nouns, real versus fictional entities, and metaphors. Even though different models focus on very different tokens, the key distinctions they make are consistent.
What this means in practice
- •For language model engineers: Trace and interpret how transformer components distinguish similar inputs, improving model debugging and analysis.
- •For natural language processing teams: Detect and understand subtle meaning differences in models’ internal representations, aiding feature design for entity or metaphor tasks.
Authors
Olli Tuomi
Abstract
Reading a transformer's internal states in token space is easy to do and hard to trust: a logit lens on a single hidden state is dominated, at intermediate layers, by the generic tokens the model would predict for almost any input. We read the difference instead. Subtracting two closely matched prompts' hidden states and projecting through the unembedding cancels the shared component and surfaces what separates them, an operation equivalent to reading a RepE/ActAdd steering vector through a logit lens. Built into a training-free tracer that reads at every position, sub-layer, and head and averages over designed baselines, it traces a compound- noun MLP->attention chain in Phi-2, confirmed there by activation patching, with the same distinction recovered across three architectures by readout and probe rather than by patching; it reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings rather than a single figurativity feature. A cross-seed control marks the boundary: across five networks differing only in initialization, the same distinction surfaces as almost entirely different tokens (top-10 overlap 0.08). What a computation looks like in token space is network-specific; the distinction it draws is not