Models use linear number representations despite curved geometry in tasks

When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Machine LearningArtificial IntelligenceComputation and Language

Summary

This paper looks at how AI models compare numbers inside their 'brains.' The authors studied a popular language model and found that although numbers can be represented in curved shapes, the model actually treats them in simpler, straight-line ways for comparing tasks. It first encodes each number as a line, combines them, and then compares parts of the combined numbers to decide which is bigger. This means AI can mix complex shapes and simple lines depending on what helps the task.

What this means in practice

  • For machine learning engineers: Improve debugging of AI models by focusing on linear representations in numerical tasks for clearer interpretability and interventions.
  • For data visualization developers: Design visualization tools that separate linear and curved data representations to better reveal how AI models process numerical information.

Authors

Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba

Abstract

One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.