Text conditioned diversity boosts image generation without retraining models
T-DIG: Text-Conditioned Diversity for Image Generation
Computer Vision and Pattern Recognition
Summary
Text-to-image tools usually show several pictures for one description, but these pictures often look too similar because models focus on quality over variety. The authors developed T-DIG, a system that uses language instructions to create diverse images by changing specific aspects like background or style, based on what users want. T-DIG guides the image generator with new prompts created by a language model, without changing the image generator itself. It works with many existing models and helps users explore different visual ideas more flexibly.
What this means in practice
- •For graphic designers: Generate varied and user-directed image versions from a single text prompt without changing the underlying image generator model.
- •For creative marketing teams: Produce multiple stylistically diverse ad images matching campaign themes by specifying variation directions in natural language.
Authors
Simran Khanuja, Rohit Gandikota, Gaurav Parmar, Graham Neubig
Abstract
Text-to-image (T2I) platforms typically return several images per prompt so users can explore different visual interpretations and choose one that matches their intent. However, T2I models are usually optimized for single-image quality which comes at a cost of multi-image diversity, limiting this exploration. Diversity is also contextual: one user wants variations in background, while another wants them in artistic style, and the same user may later want variations in clothing. The generated images should differ along whatever axis the user cares about. In this paper, we introduce T-DIG, a text-conditioned diversity framework for image generation and editing. T-DIG is an iterative framework comprising multiple modules that lets users type prompts, inspect default generations, request variations using natural language, and continue refining outputs until they're satisfied. At its core, T-DIG uses an LLM policy that produces K axis-conditioned prompts or edit instructions for a frozen generator or editor. We train this policy with GRPO using a text-conditioned diversity reward and alignment penalties. Because the image generator remains frozen, T-DIG applies to open-weight, distilled, and proprietary models. We also release the first dataset for text-conditioned prompt diversity in image generation, with the training and validation splits built from human-written axes and a test set drawn from real user interactions with T-DIG. T-DIG far exceeds baselines in improving text-conditioned diversity, while preserving base-prompt alignment across a variety of metrics and human judgements. We show that user-directed diversity exploration can be added to a wide range of image generation models, open and closed, without the need to access their weights. Website: https://image-study.org/t-dig/