Bias changes memory capacity scaling in dense associative networks
Bias-Induced Crossover in Absolute Capacity of Dense Associative Memory
Machine Learning
Summary
Memory systems in the brain and computers can store patterns, and their ability to hold many patterns is called capacity. This paper studies how having biased data, where some bits are more common than others, affects this capacity in a particular type of memory model. The authors found that bias changes how capacity grows as the system size increases, causing a crossover between two growth types depending on the bias level. They also show how to adjust the model to keep high capacity even with biased patterns.
What this means in practice
- •For neural network engineers: Design associative memory networks that maintain high storage capacity even when input data patterns are biased in their bit distributions.
- •For hardware memory architects: Develop memory devices that use polynomial interaction models optimized to adapt capacity scaling when pattern biases occur in stored data.
Authors
Yuto Sakurai, Takeaki Shimokawa, Kazunori Iwata, Kazushi Mimura
Abstract
The absolute capacity of dense associative memory has mainly been analyzed for unbiased patterns. Here we examine the effect of bias in centered binary patterns under the Krotov-Hopfield single-site criterion $P_{\mathrm{error}}=1/N$, where $P_{\mathrm{error}}$ is the probability that a single-site flip lowers the energy of a stored pattern and $N$ is the number of neurons. Each pattern component takes $1-q$ with probability $q$ and $-q$ otherwise, where $0<q\le1/2$. For polynomial interactions of order $n$, a signal-to-noise analysis gives an absolute capacity of order $N^{n-1}/\ln N$ at $q=1/2$. For fixed $q<1/2$, however, the capacity is $O(N^{n/2})$ for even $n\ge4$ and $O(N^{(n+1)/2})$ for odd $n\ge5$. For $n=3$, both the unbiased and fixed-bias capacities remain $O(N^2/\ln N)$. For $n\ge4$, these different asymptotic forms imply a nonuniform large-$N$ limit near $q=1/2$. Asymptotic matching predicts a bias-induced crossover in the region $1-2q=O(\ln N/N^{\lfloor n/2\rfloor-1})$. The crossover originates from a bias-dependent crosstalk mean that reduces the stability of sites carrying the more frequent value $-q$. Computer simulations are compared with the finite-size conditioned-Gaussian predictions. An activity-dependent control potential that cancels the conditional crosstalk mean restores the $N^{n-1}/\ln N$ capacity for fixed $0<q<1/2$ within the conditioned-Gaussian approximation.