Hopfield neural network capacity measured with correlated patterns
Capacity of the Hopfield-Hebbian dynamics with correlated patterns
Information TheoryMachine Learning
Summary
Hopfield neural networks can store and recall patterns, but usually only work well when patterns are random and independent. This paper studies what happens when the stored patterns are similar, or correlated, like real-world data often is. The authors develop a new mathematical approach to analyze how correlation affects how many patterns the network can reliably remember. They confirm that existing methods work well when patterns are only a little correlated but improve understanding in cases with stronger correlation.
What this means in practice
- •For machine learning engineers: Design and optimize associative memory systems for datasets with correlated patterns by accurately predicting network capacity.
- •For signal processing teams: Improve pattern retrieval reliability in noisy or correlated data environments using capacity estimates informed by this new analysis.
Authors
Mihailo Stojnic
Abstract
Allowing for small fraction of errors in retrieved stored binary patterns, Hopfield-Hebbian neural network (NN) associative memory dynamics [14] achieves linearly scaled capacity. The famous proportionality constant $α_c\approx 0.14$ is obtained in a large dimensional scenario as the ratio of the number of reliably stored patterns $m$ and the patterns' length $n$ under the assumption that uniformly randomly chosen binary patterns are \emph{independent} among themselves. While such an idealized scenario is now well understood, more practical scenarios with correlated/noisy datasets remaining challenging even for classical statistical mechanics replica approaches. We consider precisely such practical settings where each \emph{archetype} pattern is replaced with $M$ correlated \emph{examples} with parameter $r$ governing the quality of correlation. We develop a powerful Random Duality Theory (RDT) based framework to analyze correlated data and already on the first level of RDT lifting obtain capacity characterizations that remarkably well match behavior exhibited through simulated dynamics. Moreover, characterizations explicitly feature correlation parameters $M$ and $r$, which allows to precisely determine their effects on phase-transitioning behavior of the underlying dynamics. Our results are also a well-fitting complement to the existing literature. Namely, to circumvent challenging statistical mechanics analyses, an alternative combination of spectral and replica methods is considered in [4]. We confirm that [4]'s elegant spectral characterizations are excellent approximations in the low correlated regime. At the same time, we also close the gap in the regimes with higher correlation levels where spectral methods deviate from the practical dynamics performance.