Neural computation dimensions depend on error criteria and geometry
The Ball and the Box: Two Geometries of Computation in Superposition
Machine Learning
Summary
Sometimes computers can store more information than it seems possible, using a trick called superposition. This paper studies how many dimensions are needed to correctly compute simple yes-no logic operations from such packed information. The authors found that the number of dimensions depends on how errors are counted and that there are two different shapes—like a ball or a box—that represent these cases. They also provide exact thresholds for certain logic functions and confirm these results with simulations.
What this means in practice
- •For machine learning engineers: Design neural networks with optimal dimension size to balance error rates for logic computations using superposed features.
- •For computational neuroscientists: Model neural coding strategies that represent multiple features efficiently while maintaining reliability in decision tasks.
Tested on simulated data.
Authors
Xiaoyu Li, Lequan Lin, Dai Shi, Jiaojiao Jiang, Junbin Gao, Andi Han
Abstract
Neural representations can encode more features than they have dimensions, a phenomenon known as superposition. We study the dimension needed to compute Boolean gates from such representations. For a single threshold layer with a Gaussian random dictionary and uniformly random sparse Boolean inputs, we derive sharp dimension thresholds under two error criteria. A vanishing expected error count can require more dimensions than correctness of every output with high probability. Shared reads explain the gap: rare realizations can produce many errors at once. The expected-count threshold has ball geometry, while joint reliability has box geometry when a gate is evaluated on every feature tuple. Optimizing shared readout weights and biases gives explicit thresholds for conjunction, disjunction, and majority. For pairwise conjunction, the analysis also describes the transition near the threshold, in agreement with exact simulations.