Compressed Computation under $L^4$ Loss is likely Computation in Superposition
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how a small neural network can compute more functions than it has neurons by using a trick called computation in superposition. They trained a neural network with fewer neurons than input features and found that using a special error measure (L4 loss) helps it learn to represent and compute all features at once. They analyzed how the network does this and discovered it assigns each input a unique binary code that can be decoded to recover the outputs. The authors also showed that a small set of numbers can describe this coding scheme, and they confirmed their findings by building similar networks by hand.
neural networkssuperpositionReLU activationsparse inputsL4 lossbinary codewordpseudoinversecompressed computationsingle-hidden-layerfunction approximation
Authors
Francisco Ferreira da Silva, Stefan Heimersheim
Abstract
Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an $L^4$ loss (the mean fourth power of the error), rather than the usual $L^2$, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.