Learning systems show interference depends on use and architecture
Interference Beyond Geometry in Concept Extraction
Machine Learning
Summary
Interference in learned features is usually thought of as overlapping parts in the data representation. The authors show that this isn't just about geometry—how features interact also depends on how often and in what way they are used, plus the system's design. They identify four ways architecture controls these interactions: making features more distinct, adjusting biases, changing gains, and separating encoding and decoding. Their experiments with sparse autoencoders reveal that some architectural constraints reduce overlap while others preserve useful interactions.
What this means in practice
- •For machine learning engineers: Design sparse autoencoders to reduce destructive interference while retaining useful feature interactions through architectural constraints.
- •For signal processing developers: Improve signal representation by accounting for both geometric and statistical interference in encoded features to enhance reconstruction quality.
Authors
Valérie Costa, Bahareh Tolooshams
Abstract
Interference is commonly treated as geometric overlap between learned features. We introduce effective interference, which combines feature geometry and code statistics to capture realized interactions, distinguishing constructive from destructive interference and frequent weak interactions from rare strong ones. Under local fixed-support assumptions, we characterize how architectural constraints shape interference through four mechanisms: feature orthogonalization, bias compensation, gain adaptation, and encoder-decoder separation. Experiments with sparse autoencoders show that constrained architectures selectively reduce overlap among co-active features, while bias, gain, and encoder freedom allow constructive cross-contributions to remain. Together, these results show that interference in learned representations depends not only on feature geometry, but also on how features are used and on the architecture that produces their codes.