Self supervised signals miss encoder decline in image training
Improving Signals, Declining Encoder: A Known-Answer Test of Label-Free Monitoring in Self-Supervised Learning
Machine LearningComputer Vision and Pattern Recognition
Summary
Sometimes, when computers teach themselves to recognize images, parts of what they learned can suddenly get worse without showing up in usual checks. The authors trained a system on fake images where they knew exactly what features mattered and found that even though the quality dropped drastically, the signals people normally use to watch training didn’t notice. Most checks and losses kept improving even when the system was losing knowledge. This means current ways to watch self-learning might miss when the system starts to forget.
What this means in practice
- •For machine learning engineers: Improve monitoring tools to detect when self-supervised models lose learned features despite loss improvements.
- •For automated image analysis teams: Identify risks of undetected decline in feature quality during training of unsupervised image encoders.
Tested on simulated data.
Authors
Nilotpal Kakati
Abstract
Self-supervised training is monitored without labels, through the loss and statistics of the learned features. We report a case where we know the encoders lost part of what they had learned, so we can check whether these signals noticed. We trained LeJEPA encoders on synthetic images with known generating factors. In nine of ten seeds, the readability of those factors fell abruptly and stayed down, nearly always along the same direction in factor space. Meanwhile the loss kept falling, and most of the eleven label-free signals we tracked kept improving and rated the declining runs better than runs that did not decline. Standard change-point detection rarely flagged the decline near its onset, and selecting checkpoints by these signals mostly chose ones after it.