Balancing image data sets by generating under-represented samples
Balancing Representations by Identifying and Generating Under-represented Data
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
When computers learn from large collections of pictures without labels, they mostly understand common images well but struggle with rare ones. The authors present a way to identify which rare images are lacking and create new similar images using a special generator, improving the computer’s overall learning. This approach works without needing labels and helps the computer learn better from unbalanced and messy photo collections. Their method boosts performance in several image recognition tasks and works with different learning techniques.
What this means in practice
- •For computer vision engineers: Improve image representation quality in systems trained on large, uncurated web photo datasets by augmenting under-represented image types without labels.
- •For autonomous vehicle developers: Enhance recognition of rare object categories in onboard perception systems by generating synthetic examples during model pretraining.
Authors
Filipe Laitenberger, Hirokatsu Kataoka, Yuki Asano, Cees Snoek, Mohammadreza Salehi
Abstract
Self-supervised learning inherits the coverage of its source data. When an unlabeled corpus is long-tailed, the representation becomes dense around frequent modes and thin where coverage is weak. Objectives for long-tailed SSL reweight the images already in the corpus and add no new ones. We introduce BRIDGE (Balancing Representations by Identifying and Generating Under-represented Data), a label-free feedback loop around the source set. The current encoder scores every source image by its mean kNN distance, and a budgeted acquisition rule selects images from a band of that score distribution where support is thin and the generator still produces faithful variants. A frozen image-to-image diffusion model then adds local variants of the selected anchors, and pretraining continues on the enlarged source set. We formulate acquisition as recoverability-weighted allocation of local support and derive when a finite budget should favor a sparse band that excludes the extreme tail and when reallocation after each repair is optimal under a fixed objective. Across five long-tailed, web-scraped and synthetic source regimes, BRIDGE has the highest seven-task linear-probe and kNN average of the compared methods, and paired experiments extend the gain to ViT-S under contrastive and self-distillation objectives, to low-label fine-tuning and to semantic segmentation. The linear-probe gain is smaller under masked reconstruction. Generated repair matches an oracle that restores the withheld real images, and under a frozen encoder the inserted images shrink the neighborhoods of the selected anchors about twice as much as those of matched unselected images. BRIDGE needs no labels and leaves the pretraining objective unchanged, so it can be added to pretraining on uncurated web data, where the encoder decides which images to generate.