AI summaryⓘ
The authors create a framework to understand how features and their relationships behave inside deep neural networks, especially focusing on how different mathematical parts work together or clash. They measure incompatibilities between certain network components using special mathematical objects called commutators and break down these incompatibilities into understandable sources. Their work shows that good alignment of features in a network isn’t always a straightforward result of training but depends on complex interactions like transport and cancellation effects across layers. Through theory and tests, they identify conditions under which these interactions stabilize or decay, providing insight into why some training behaviors occur.
Deep neural networksFeature geometryCovarianceGatesBackward sensitivitiesCommutatorsGradient flowSpectral alignmentLyapunov principlesTransport phenomena
Abstract
We develop a finite-width geometric framework describing how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. Incompatibility among weight-generated covariance, gates, and backward sensitivities is quantified through three families of commutators: between gates and covariance, between sensitivities and covariance, and between average gradient outer products (AGOPs) and neural feature matrices (NFMs). An exact layerwise identity decomposes the sensitivity-covariance commutator into four sources: downstream transport, adjacent-layer imbalance, pointwise sensitivity fluctuations, and nonlinear gate-covariance interactions. The AGOP-NFM commutator is a singular-value-weighted transport of the internal commutator, explaining why observed feature-side alignment alone does not determine the internal geometry from which it emerges. Buffered localized energies resolve mixing between separated covariance subspaces. We establish spectral-gap, projector-evolution, and stabilization estimates, and formulate conditional Lyapunov principles that yield decay under explicit geometric error-bound or intrinsic-damping assumptions. These criteria do not follow from gradient flow alone and clarify why risk reduction need not imply commutator collapse. Analytic examples and numerical experiments exhibit factorization of spectral and activation geometry, transient growth, and cancellation among nonzero sources. In tested finite-time regimes, cancellation dominated by a negative transport-imbalance interaction persists across depths, widths, and two regression benchmarks. Spectral alignment therefore appears as a layer- and scale-dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.