Modern hopfield models reveal how hierarchical prototypes form and generalize
Hierarchical Prototype Emergence in Modern Hopfield Models
Artificial IntelligenceMachine Learning
Summary
Many kinds of data have groups within groups, like families inside cities inside countries, but it is hard for memory systems to learn and use these groupings well. The authors study a type of memory network called a Hopfield network that uses polynomial activation functions, and show how it can store and recall patterns that form hierarchical groups. They mathematically find when each group in this hierarchy remains stable, meaning the network remembers them reliably. They also show it requires only a small amount of extra information to imagine new patterns beyond those it has seen or groups it has learned, using a prototype-based approach.
What this means in practice
- •For machine learning engineers: Design memory networks that efficiently learn and generalize hierarchical patterns in image data.
- •For computer vision developers: Improve image generation systems by incorporating stable hierarchical memory prototypes for more coherent outputs.
Tested on one dataset.
Authors
Aditya Cowsik, Adithya Sriram
Abstract
Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.