Context Is King: How In-Context Specification Shapes the Geometry of Concepts
2026-07-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors found that large language models can arrange concepts like weekdays or months in different geometric patterns, like circles or trees, based on instructions given in the current context. This means the model doesn't just rely on fixed learned structures but can change how it represents information on demand, even with nonsense tokens. In bigger models, this context-driven structure becomes clearer and actually influences the model's answers, while smaller models show weaker effects. The study shows the geometry the model uses depends on what the context specifies rather than only on prior training.
large language modelsmanifoldsin-context learninggeometry of conceptstopologyactivation patchingrepresentational similaritymodel scalecontextual modulation
Authors
Elad David, Max Fomin
Abstract
Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, usually taken to be a fixed world-model the network stores and looks up. We show that context is king: the structure a model actually uses is set by the in-context specification. A declarative rule fixes not only which relations the geometry encodes but its topology type: the same tokens form a cycle or a branching tree on command, built even on arbitrary, meaning-free tokens with no prior to inherit, which a relabeled stored shape cannot do. When the specification conflicts with a strong pretrained prior, the context-set geometry dominates it in capable models, read from the same activations (representational similarity 0.6--0.9 to the imposed structure versus near-zero to the prior), across the priors we test and both families we study (Gemma, Qwen). Activation patching shows the map is causally used, not a probe correlate: swapping one entity's activation for another's makes the model answer with the other entity's successor under the imposed order. A rough map forms readily, present even in small and base models; what scale gates is using it cleanly: clean dominance and the causal crossover emerge only in the larger models (up to Gemma-31B and Qwen-27B) and weaken or reverse below, so a mechanism present in a large model can be absent in a smaller one of the same family. Whether the model builds this geometry anew or reconfigures a stored one we leave open; operationally, the geometry it uses is the one the context specifies.