SSMs and transformers use different spaces but encode concepts similarly
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
Computation and Language
Summary
Different AI language models, like transformers and state-space models (SSMs), have different internal structures. The authors found that while transformers focus their information in one main direction inside their 'brain,' SSMs spread it out evenly. Despite these differences, both still store knowledge in similarly sized ways and group ideas in very similar local patterns. This means their overall function in understanding language is surprisingly alike, even if their inner workings look different.
What this means in practice
- •For machine learning engineers: Design hybrid language models that combine SSM and transformer layers to optimize internal representation efficiency without losing conceptual encoding power.
- •For natural language processing teams: Improve model compression methods by understanding that both SSMs and transformers have similar effective capacities despite differing geometric properties.
Authors
Amit Ben-Artzy, Roy Schwartz
Abstract
Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers despite relying on fundamentally different architectures. This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations? We study this question through a multi-scale analysis of representations in transformers, SSMs, and hybrid architecture. First, we find that SSMs distribute their representational information evenly across all dimensions, whereas transformer representations are heavily dominated by a single principal direction. By evaluating hybrid architectures, we observe that the representation space becomes increasingly skewed toward a single dominant direction after each attention layer. Next, we explore how the different geometric spread of representations impacts representational capacity through compressibility. Surprisingly, we find that despite their contrasting geometric structures, both architectures exhibit tightly matched effective capacities. We further investigate whether this skewed geometry affects how concepts are encoded. Using rank-constrained probes, we demonstrate that both architectures encode concepts in subspaces of surprisingly similar dimensionality. Furthermore, we demonstrate that the transformers' dominant principal direction does not inherently encode more conceptual information. Finally, we zoom in and examine the alignment between manifolds, either by analyzing representations of specific topics or by looking at the nearest neighborhoods of tokens, and find that they are highly aligned. Ultimately, our analysis suggests that while transformers and SSMs induce different usage of latent space, they display a striking functional convergence at the level of local semantic manifolds.