Functional Degeneracy in Neural Networks: Measurement and Pruning
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how much a machine learning model can be made smaller without changing how it works. They introduce a new measure called behavioral recovery rank, which counts how many key directions in the model's behavior you need to keep to keep the model’s performance. Their findings show that pruning methods that remove parts of the model by structure or size keep more important information than expected, even when the model has learned enough for the task. This means the model's important info is spread out over many parts, not just single weights or neurons.
model compressionmachine learningbehavioral recovery rankbehavioral-Hessian eigendirectionsstructural pruningmagnitude pruningfunctional redundancymodel performanceparameter directionsneural networks
Authors
Maria Matveev, Pascal Esser, Ayush Bharadwaj, Lucius Bushnaq, Gitta Kutyniok
Abstract
A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.