Neural networks reveal true data factors when dimensions match or constraints apply
Complete or Sparse: A Tale of Two Identifiabilities
Machine Learning
Summary
This paper explores when neural networks can uncover the hidden building blocks behind data instead of mixing them up. The authors find that if the network has enough room (dimensions) to represent the hidden factors, it learns simple combinations. If not, it mixes things in complicated ways. They also show that adding certain rules, like keeping signals positive or sparse, helps the network identify each hidden factor clearly. Their findings unify ideas from different fields and work on both simulated data and real brain and AI models.
What this means in practice
- •For neural network developers: Design neural network architectures with appropriate dimension or constraints to ensure interpretable hidden representations.
- •For computational neuroscientists: Use the identified conditions to interpret brain activity patterns more precisely by linking network units to individual neurons.
Authors
Junze Zhou, David Klindt
Abstract
Understanding when neural networks learn interpretable representations is central to mechanistic interpretability. We study this through identifiability: when can supervised learning recover the true latent variables underlying the data? We introduce two data generating processes distinguished by their causal direction where either labels cause latents (generative) or latents cause labels (discriminative). When the representation has at least as many dimensions as the latent space, we prove that the optimal encoders are linear in the latents. Conversely, when dimensions are insufficient, identifiability generically fails and encoders become nonlinear. We then show that additional constraints yield recovery of individual latent variables, not just linear mixtures. In the generative setting, non-negative activations and energy efficiency force each neuron to encode exactly one latent, providing a precise account of selective grandmother cell-like responses. In the discriminative setting, when latents are sparse and the true measurement matrix satisfies the spark condition, the learned encoder inherits compressed sensing structure, enabling sparse dictionary learning to perfectly recover individual latents. This constitutes, to our knowledge, the first identifiability result for undercomplete nonlinear independent component analysis. Our framework unifies nonlinear ICA, compressed sensing, and neural network interpretability, delineating exactly when representations transition from nonlinear distortions to linear representations and, under additional constraints or steps, to clean, axis-aligned codes. Theoretical predictions are validated on synthetic benchmarks and on activations drawn from vision models, large language models, and macaque inferotemporal cortex.