AI summaryⓘ
The authors argue that simply measuring how close network representations are isn't enough to understand their behavior because similar internal states can act differently and vice versa. They propose making these representations 'volume-based' by adding controlled noise, which helps relate similarity to actual behavior through statistical measures. They implement this idea by adjusting transformer models to treat their internal states with finite precision influenced by a global budget, allowing them to track how information changes layer by layer. Their experiments with vision and language models show that this approach highlights how certain distinctions are preserved or lost, offering a new way to interpret transformers grounded in how they process information. This method adds a functional perspective to understanding transformer computations beyond existing tools.
Representational similarityTransformer modelsLayerNormIsotropic Gaussian noiseBhattacharyya coefficientInformation theoryData-processing inequalityDistillation fine-tuningResidual streamAttention heads
Abstract
Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied to downstream function: nearby states may produce different behaviors, while distant states may behave similarly. We instead give representations volume, turning similarity into statistical distinguishability. Overlapping stochastic representations necessarily induce overlapping downstream distributions, grounding latent comparison in model function and bringing it under information-theoretic tools such as the data-processing inequality. We realize this idea in pretrained transformers through a light-touch modification to LayerNorm: at each residual-stream read, we normalize the state, add isotropic Gaussian noise, and renormalize. During distillation fine-tuning, one learned allocation parameter per residual-stream read distributes a fixed global rate budget across the processing stack. The resulting model can be viewed as transformer blocks reading the residual stream with learned finite precision under a shared global rate budget. Using the Bhattacharyya coefficient, we trace which counterfactual distinctions are preserved through MLP blocks or selectively exposed to the query, key, and value computations of individual attention heads. Experiments on ViT-S and GPT-2 small reveal the depthwise propagation of continuous visual perturbations and head-specific sensitivity to token distinctions aligned with known attention motifs. These results establish distinguishability as a functionally grounded lens on transformer computation that complements existing interpretability approaches.