Expressivity of congruence-based architectures for DNNs on positive-definite matrices
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a type of neural network layer used to work with special matrices called symmetric positive-definite matrices. These layers transform the input matrix using a weight matrix and its transpose. They find that restricting this weight matrix to be semi-orthogonal reduces the network's ability to learn complex patterns, effectively simplifying the model too much for some activation functions. This limitation is linked to a mathematical theorem about eigenvalues called Poincaré's separation theorem. The authors also explore different ways to classify the resulting features, comparing methods that respect the geometry of these matrices.
symmetric positive-definite matrixneural networkscongruence transformationsemi-orthogonalitySPDNetactivation functionPoincaré's separation theoremspectral diversityRiemannian classifierdimensionality reduction
Authors
Antonin Oswald, Estelle Massart
Abstract
This work studies neural architectures for classifying symmetric positive-definite matrices, focusing on congruence-like layers, in which the input matrix is multiplied on the left and right by a (possibly rectangular) weight matrix $W$ and its transpose. Such layers lie at the core of the celebrated SPDNet and have also been employed independently for dimensionality reduction on positive-definite data. We show that the (semi)-orthogonality constraint commonly imposed on $W$ limits the expressivity of these layers: for certain activation functions, the resulting architecture collapses to a one-hidden-layer equivalent. This lack of expressivity follows from a loss of spectral diversity in congruence-like layers for semi-orthogonal $W$ and is a direct consequence of Poincaré's separation theorem. We then examine the choice of the final classifier, comparing several Riemannian classifiers and discussing their compatibility with the feature maps produced by congruence-like layers.