Large language models reveal complex non-linear concept structures
Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
Machine LearningComputation and Language
Summary
Large language models process information in ways that cannot be fully explained by simple straight-line (linear) assumptions. This paper shows that concepts inside these models are actually spread out along curved, multi-dimensional shapes called non-linear manifolds. The authors develop a new way to measure how these concept shapes align across different parts of a model and between different models or training stages. Their findings show patterns in how models handle language syntax and meaning evolve across layers and vary with training and model family.
What this means in practice
- •For language model engineers: Improve model interpretability by identifying complex concept structures beyond linear assumptions to better debug and optimize LLM behavior layer by layer.
- •For multilingual ai developers: Tailor multilingual LLM training and fine-tuning strategies recognizing that concept sharing across languages depends on model family and training specifics.
Authors
Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff
Abstract
Understanding information processing in large language models (LLMs) requires dissecting the geometric organization of their internal token representations. While existing mechanistic interpretability (MI) methods seek to extract concepts, they are constrained by a strong linearity assumption challenged by evidence of non-linear feature manifolds. We move beyond linear concepts by adapting Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from computer vision to token-level LLM activations, modeling concepts as low-dimensional manifolds. To compare concept manifolds across layers and models, we introduce a concept-based alignment (CBA) score, a generalized Rand index that measures geometric proximity without explicit feature matching. Our analysis yields six key findings: (i) a neighboring-layer sanity check shows CBA is more sensitive than PCA- or CKA-based linear baselines; (ii) layer-by-layer alignment matrices reveal two block structures in intermediate and late layers, consistent across models and obscured by linear metrics; (iii) concept composition remains syntax-dominated through most of the network before giving way to increasingly mixed syntactic-semantic concepts in later layers, with increasing output-orientation toward the final layers; (iv) multilingual concept sharing between English and Mandarin is training-dependent rather than universal, strongest in Qwen, weaker in Llama, and absent in GPT-2; (v) inter-model alignment mirrors this structure, with strong correspondence between same-family Qwen models of different scale but weak alignment across model families; and (vi) across Tulu-3 training stages, alignment is highest between adjacent stages, with the largest shift between the base model and SFT, while subsequent preference-alignment stages (DPO, RLVR) leave early layers largely unchanged and RLVR mostly preserves DPO's concepts in late layers.