AI summaryⓘ
The authors study how neural networks learn to represent concepts as consistent directions in their internal data space during training, a process they call "abstraction." They find exact mathematical descriptions of this process in simple linear networks, showing that the shape of data, network depth, and initial settings affect how well concepts form. Extending to more complex nonlinear networks, they show different activation functions influence this process differently and that abstraction gets weaker in activations compared to earlier computations. They also find evidence for their theory in real models and use it to improve how well simple tools can read concepts from large language models. Overall, their work explains how abstract concept directions develop over time inside neural networks.
linear representation hypothesisabstraction dynamicslinear probesconcept directionsneural network trainingnetwork depthinitialization scalenonlinear activationsReLUlinear probes generalization
Authors
William W. Yang, Andrew M. Saxe, Peter E. Latham
Abstract
In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space. In deep learning, this idea is known as the linear representation hypothesis and underpins many interpretability and control methods based on linear probes, from concept detection to activation steering. Yet while prior work has studied whether such directions should exist $\textit{after}$ training, the dynamics of how they emerge $\textit{during}$ training remain poorly understood. Here, we develop a framework to study the alignment of concept directions during training - a process we call "abstraction". In a minimal linear network setting, we obtain exact solutions for the full trajectory of abstraction. These solutions reveal key analytic principles governing abstraction: (i) data and target geometry jointly determine abstraction at the end-of-learning, (ii) abstraction improves with network depth, and (iii) initialization scale controls the maximum abstraction reached during training. Extending our theory to nonlinear networks, we analyze how the choice of nonlinearity affects abstraction dynamics: erf networks approximate the linear theory, while abstraction in ReLU networks depends less on target geometry and more on input geometry. Across both, we prove a striking attenuation law: both nonlinearities weaken abstraction in activations relative to preactivations. We find evidence for this law in open models (DINOv3, Gemma 4) and apply our theory to improve linear probe generalization in LLMs. Together, our results provide a dynamical theory of abstraction with implications for interpretability and control.