Spatial dependencies improve sparse autoencoders for visual concepts
The Independence Prior of SAEs Fragments Visual Concepts
Artificial Intelligence
Summary
Sparse autoencoders are computer models that break down images into simple building blocks, but they often assume these blocks appear independently in different parts of the image. The authors found that this independence assumption doesn’t match how images naturally work, since nearby parts of an image usually relate to each other. They introduced a new method, Spatial-SAE, that takes into account these spatial relationships between parts. This new approach better recovers and explains the building blocks of images compared to standard methods.
What this means in practice
- •For computer vision engineers: Improve the interpretability of visual feature extractors by incorporating spatial dependencies between image parts.
- •For machine learning practitioners: Enhance unsupervised concept discovery within image representations by using spatially-aware sparse autoencoders.
Authors
Tommaso Mencattini, Giorgos Nikolaou, Donato Crisostomi, Thomas Fel, Francesco Montagna, Emanuele Rodolà, Francesco Locatello
Abstract
Sparse Autoencoders (SAEs) decompose model activations into sparse combinations of interpretable dictionary atoms. Although SAEs are grounded in the Linear Representation Hypothesis (LRH), their objective smuggles in an additional prior: concepts across patches are treated as independent, an assumption clearly violated by natural images and by the activations they induce. We therefore specialize LRH to vision through the Markov-Field Linear Representation Hypothesis (MFLRH), which adds the missing spatial dependencies to the LRH assumptions. We thus propose Spatial-SAE as an amortized MAP estimator under the MFLRH. Spatial-SAE consistently outperforms standard SAEs in concept recovery and interpretability. Across four variants, it achieves a 96% average win rate on synthetic concept recovery and improves interpretability on DINOv2 activations, at a reconstruction cost concentrated in high spatial frequencies.