Hidden states of large language models show tree like distance patterns
A Hyperbolicity Atlas of Large Language Model Hidden States
Computation and Language
Summary
Large language models (LLMs) produce many hidden numerical codes inside them that help generate responses. This study explores whether these codes form a pattern like branches on a tree when looking at how far apart they are. The researchers found that certain layers inside the models show stronger tree-like patterns, especially the middle and final layers. They also found these patterns change depending on the model type, layer depth, and the kind of questions asked. This helps explain how these models organize information internally and suggests ways to analyze their behavior better.
Large language modelHidden stateGromov HyperbolicityHierarchical structureTree-like distanceLayer depthModel scaleDistance measurePrompt tokensModel specialization
Authors
Zhichao Yang, Yuanze Hu, Gen Li, Qingchen Yu, Shiying Duan, Xinyu Wang, Ye Qiu, Zeming Liu, Guangxu Chen, Zhaoxin Fan
Abstract
LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.