Salient factor discovery improves subtype recognition and generation accuracy
SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations
Computer Vision and Pattern RecognitionMachine Learning
Summary
When given two groups of images, like faces with and without glasses, the authors’ method finds specific differences that stand out only in the group with glasses. This way, the method learns details about types of glasses without needing labels for those types. It can then generate new images showing these different glasses styles. Their approach also works for medical images, distinguishing diseases using only basic labels.
What this means in practice
- •For image generation developers: Generate new, varied images emphasizing subtle but meaningful differences found in example images without subtype labels.
- •For medical imaging teams: Identify and separate diseases in retinal scans using label-light learning of salient disease features.
Authors
Shuang Liang, Lejun Liao, Shiyuan Zhang, Max C. Zhang, Xiaolong Luo, Han Wang, Stefano Anzellotti, Yuan Yuan
Abstract
Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates \textit{salient} factors specific to the target from \textit{common} content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description. We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity \textit{reconstruction} (rFID below $2$) with unsupervised \textit{subtype discovery}, recovering the digits better than baselines (probe accuracy $0.950$ vs.\ at most $0.281$) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned \textit{generation} raises Digits-ImageNet subtype accuracy over the unfactorized latent ($90.5\%$ vs.\ $27.7\%$) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.