Spectral objectives improve image feature learning and classification accuracy
Compositional Objectives: Learning Structure in Structure
Machine Learning
Summary
This paper looks at how computer programs can learn to recognize and organize parts of images better. The authors study a technique to measure how much useful structure a program can learn from images, focusing on spreading out information rather than concentrating it. They show that when programs look at only parts of images, they learn features that help them classify images much better. They also find that making the program predict relationships between parts makes it better at understanding details but worse at simple classification.
What this means in practice
- •For computer vision engineers: Improve image classification models by using spectral objectives over masked image parts for stronger feature representations.
- •For machine learning model trainers: Enhance feature learning in deep networks by adjusting what input parts the observer focuses on, boosting downstream task performance.
Authors
Pranavchandra Vivekananda, Sumukh Bettadapura, Ajan Subramanian
Abstract
Intelligence is defined in many ways. One of these definitions defines intelligence as the pursuit of learnable novelty. However, learnable novelty can be meaningless without the ability to compose the learned structures to take action and achieve goals. Learnable novelty builds on epiplexity, which is a way to measure learnable structure in data through a bounded observer. In this paper, we investigate a closed-form spectral approximation to compute epiplexity. We use a fixed-trace constraint and find that the epiplexity objective prefers a more uniform distribution of spectral mass rather than concentrating it in a small number of directions. However, a representation may spread information across many directions without organizing that information into features useful for a particular task. To address this gap, we propose a compositional objective whose observer measures the relationships between the parts and interactions of an image. We compare it with the original spectral objective given only the masked parts. In our ImageNet training runs, the spectral objective with masked parts produces an almost maximally spread representation while achieving the strongest frozen-feature classification performance on most evaluations, more than doubling the linear-probe accuracy of the whole-image baseline. Across multiple image benchmarks, changing what the observer sees matters more than adding relation and interaction tokens. At the same time, our prediction-oriented compositional objective produces substantially better held-out observer prediction but relatively weaker classification, revealing that spectral diversity, predictability, and downstream utility are distinct properties. These results suggest that the usefulness of spectral spreading depends not only on how much structure is preserved, but on which relationships the observer makes available to the objective.