A Comparative Study of Label-free Representation Quality Metrics in Deep Learning
2026-08-24 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors studied different ways to judge how good deep neural network features are without using any labels. They grouped these methods into three categories and explored how they relate to each other. By testing many models on varied tasks, they found that one measure called intrinsic dimensionality (ID) predicts performance best. However, the usefulness of all measures changes depending on the type of neural network and how it was trained. Their work helps clarify when and how these label-free metrics should be trusted.
deep neural networkslabel-free metricsintrinsic dimensionalityrepresentation qualityspectral metricsdownstream task accuracyarchitecture classtraining objectiveobject classificationscene recognition
Authors
Daniel Richards Arputharaj, Daniel Jönsson, Gabriel Eilertsen
Abstract
We present a comparative study of label-free metrics for assessing the quality of representations in deep neural networks to understand their reliability under a wide variety of configurations. We group existing label-free metrics into three families based on their construction and analytically establish connections between metrics within the same family. We then characterise the sensitivity of spectral metrics through controlled synthetic experiments. Finally, all label-free metrics are evaluated against downstream task accuracy across a diverse set of 260 vision models on six datasets spanning generic object classification, fine-grained object classification, scene recognition and geospatial task, stratifying results by architecture class and training objective. We find that intrinsic dimensionality (ID) is the most reliable predictor among the metrics considered. However, the reliability of all metrics, including ID, is moderated by architecture class and training objective. Our results provide a clearer understanding of what label-free representation quality metrics measure, when they are reliable, and how to interpret them in practice.