Determinant maximization improves diversity in text to image diffusion sampling
Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling
Computer Vision and Pattern Recognition
Summary
Text-to-image models often generate very similar images when guided strongly, which limits their usefulness for tasks needing a wide variety of outputs, like data augmentation. The authors found that just maximizing differences between pairs of images isn't enough to ensure true diversity across the whole batch. They propose a new method called DetNO that encourages the batch of images to cover a larger range by maximizing a mathematical measure of volume in the noise space used to generate images. DetNO improves diversity without losing image quality and leads to better performance when using generated images for training other models.
What this means in practice
- •For machine learning engineers: Generate more diverse synthetic images for training AI models to improve accuracy, especially in low-data situations.
- •For computer graphics developers: Create richer varied image batches for style transfer and creative content generation using text-to-image diffusion.
Authors
Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Sungho Koh, Dong-Jin Kim
Abstract
Strong guidance in text-to-image (T2I) diffusion models concentrates sampling toward high-density modes of the conditional distribution, collapsing independently sampled noise vectors onto nearly identical visual prototypes. This limits utility for tasks that demand distributional breadth, most critically synthetic data augmentation. Diversity, however, is a collective property of the whole batch: a set can preserve every pairwise distance yet still collapse onto a low-dimensional subspace. Existing sum-of-pairwise noise-optimization and repulsion objectives respond only weakly to this higher-order failure. We propose DetNO (Determinant Noise Optimization), which formulates diverse generation as RKHS volume maximization on the initial noise, maximizing the log-determinant of an RBF Gram matrix over one-step Tweedie predictions. This set-level functional equals the squared batch volume; its repulsion is strongest when the batch is most collapsed and self-attenuates as diversity is achieved. An anchor-based fidelity term keeps the optimized noises near plausible regions, and Lagrangian relaxation gives a two-term objective with a single hyperparameter $λ$ governing the diversity-quality trade-off, without training or sampler modification. Across three backbones (SD1.5, SDXL, SD3), DetNO achieves leading intra-batch diversity in VendiScore and Mean Self-Similarity, and a downstream few-shot classification experiment confirms that broader mode coverage brings statistically significant accuracy gains for synthetic data augmentation in the low-shot regime. Fixing the pipeline and varying only the batch objective, both RBF-based aggregators outperform InfoNCE, and the log-determinant attains the best downstream accuracy on both benchmarks. Its diversity advantage over InfoNCE widens as the batch grows. Code is available at https://github.com/mobled37/DetNO.