Knowledge matrices reveal deeper network input views beyond hidden activations
Hidden Activations are not Enough I: Knowledge Matrices as Higher Representations
Machine LearningNeural and Evolutionary Computing
Summary
Neural networks process inputs to make decisions, but understanding precisely what they 'know' inside can be tricky. This paper studies a special matrix called the knowledge matrix that captures the network's representation of an input more fully than just looking at hidden layers. The authors show this matrix relates closely to how the network’s outputs change with inputs and can help measure differences between networks and analyze attacks. They also show that simply looking at hidden activations is not enough to capture all this deeper knowledge.
What this means in practice
- •For machine learning engineers: Use knowledge matrices to better evaluate and compare neural network behavior beyond hidden layer outputs, improving model interpretability and robustness analysis.
- •For security analysts: Employ knowledge matrix geometry to detect and quantify adversarial input changes in neural networks, aiding in vulnerability assessment.
Authors
Marco Armenta
Abstract
We study the knowledge matrix of a trained feedforward network as a higher representation of its inputs. A network is a pair $(W,f)$, a thin representation $W$ of its quiver and an activation $f$; its function factorizes through the space of quiver representations, each input $x$ inducing a representation, and the knowledge matrix $M(x)\in\mathbb{R}^{C\times(d+1)}$ is the contraction of that representation to one matrix whose rows sum exactly to the logits. At one trained network we ask what determines it, what it is invariant to, what it determines, and what its geometry measures. Under (LCS), a locally constant slope diagonal, as for ReLU, the matrix at a regular input is a function of the realized germ; its stabilizer among encodings regular there is exactly the germ stabilizer at inputs with no vanishing coordinate, neuron permutation a special case; and it recovers the germ, whereas hidden activations, gauge-covariant and germ-incomplete, are not enough. Under (LCS) it equals per-class gradient$\times$input plus an exact aggregate bias attribution, grounding it in attribution theory and computing it by $C$ vector-Jacobian products instead of probing. The fixed shape gives an alignment-free per-sample distance between ResNet-152, DenseNet-121 and GoogLeNet; the row-sum identity gives an exact visible/invisible displacement decomposition whose unit-free coherence $A=(d_Ψ/d_M)^2$ puts adversarial germ motion at median $A\le 0.23$, with an attack-family ordering concordant across six architectures (Kendall $W=0.921$; $0.97$ on the three networks at full scale). Two honest negatives: on AlexNet/CIFAR-10 penultimate features win 5 of 6 detectors and all 16 attacks, and a matrix-direction counterfactual fails 0/54.