Large language model layers show varying tree-like structures in hidden states
A Hyperbolicity Atlas of Large Language Model Hidden States
Computation and Language
Summary
The paper studies how the hidden layers inside large language models arrange information like a tree, even though the data is stored as regular vectors. The researchers measured this tree-like quality across different model sizes, layers, types, and tasks using a special mathematical way called Gromov Hyperbolicity. They found that the middle and final layers often show more clear tree structures, while size has less impact. This helps reveal how models organize knowledge internally and which parts might need closer study.
What this means in practice
- •For llm engineers: Improve model analysis by identifying which layers and tasks produce hierarchical information structures using hyperbolicity measures.
- •For nlp system builders: Select model layers showing stronger tree-like hidden states for tasks relying on hierarchical reasoning or structure.
Authors
Zhichao Yang, Yuanze Hu, Gen Li, Qingchen Yu, Shiying Duan, Xinyu Wang, Ye Qiu, Zeming Liu, Guangxu Chen, Zhaoxin Fan
Abstract
LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.