Abstract representational geometry supports inference in large language models
2026-06-22 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how large language models (LLMs) and humans adapt to changing tasks by learning hidden rules from limited information. They found that while LLMs are less consistent than humans in making these flexible inferences, when they do, LLMs form internal patterns similar to brain structures called the hippocampus. These patterns are organized in layers, with deeper layers showing more abstract reasoning-like structures. The researchers also demonstrated that encouraging certain geometric patterns in the models can improve their reasoning abilities. Overall, the study suggests that abstract geometric representations inside LLMs help support flexible inference.
hippocampuslarge language modelsabstract representationneural state spacecontextual reversal learningrepresentational geometryinferencemodel layersgeometric regularizationtask adaptation
Authors
Yunan Zeng, Yuwang Wang
Abstract
A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.