Disentanglement with Holographic Reduced Representations
2026-06-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address the challenge of disentangling different factors in data by using neural networks in a new way. Instead of representing factors as continuous values, they treat them as symbolic structures and use holographic reduced representations (HRR) to keep things differentiable. Their method helps separate factors effectively and is competitive with existing techniques, while also being more robust to noise. They back their approach with theoretical analysis showing how HRR naturally encourages independent symbol-value pairs and limits how many concepts can be encoded. This leads to latent representations that sum vectors rather than use scalar dimensions, differing from traditional autoencoders.
DisentanglementNeural networksHolographic reduced representations (HRR)Variational autoencodersGenerative adversarial networksSymbolic representationsDifferentiabilityInformation-theoretic constraintsLatent spaceNoise robustness
Authors
Jhonny J. Velasquez Olivera, Christo K. Thomas, Walid Saad
Abstract
Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning. Prior work has addressed this problem with variational autoencoders and generative adversarial networks that incorporate ideas from variational inference and information-theoretic constraints. In contrast to methods that rely on continuous representations, we propose a design that treats disentangled representations as symbolic structures, motivated by the compositional relationships among the concepts that make up samples from a distribution. However, learning discrete symbolic structures with neural networks while maintaining differentiability is difficult and often requires complex architectures. To address this, we introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement. We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines, as measured by latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts can be reliably encoded, giving a quantitative account of the inductive bias toward disentanglement. The resulting representations differ from standard autoencoder-based models, in that their latent units are vectors that are summed together, rather than scalar dimensions of a low-dimensional latent vector. We show that this HRR representation is more robust to noise than other disentangled representations and maintains reconstruction quality across a range of SNRs.