Text-to-image model steers demographics of each person separately
Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation
Computer Vision and Pattern Recognition
Summary
Text-to-image AI models often create pictures that reflect societal biases, like stereotypes about jobs or races. The authors show that common fixes apply the same adjustment to the whole picture, which makes all the people in the image look similar, losing variety. They propose a new way called Spatially Gated Steering that adjusts each person in the scene independently by detecting them inside the AI model itself. This method improves diversity within each image while keeping overall fairness and good picture quality without needing to retrain the model or provide extra layout information.
What this means in practice
- •For ai developers: Generate images with multiple people showing diverse demographics individually without retraining the model.
- •For advertising agencies: Create more diverse and balanced multi-person images in campaigns without sacrificing image quality.$Commercial implications: Enables production of fairer marketing visuals appealing to diverse audiences, enhancing product reach and brand image.
Authors
Venkatesh Thirugnana Sambandham, Torsten Schon
Abstract
Text-to-Image models reproduce societal stereotypes, and existing mitigation applies one demographic correction to the whole scene. In multi-person scene generation this creates a trade-off: global steering improves aggregate balance only by homogenizing the people within an image. Steering-based methods like Embedding Arithmetic drive the intra-image diversity below the unsteered baseline. Text- and latent-projection approaches shift the aggregate skew with only marginal gains in intra-image diversity. We present Spatially Gated Steering (SGS), which steers the demographics of each person independently by extracting per-person masks from the Diffusion Transformer's (DiT) internal attention maps at a critical denoising timestep and applying a distinct demographic vector within each mask for the remaining steps. SGS needs no fine-tuning, inversion or layout specifications, and uses a lightweight segmentor over frozen DiT attention features to localize subjects at early timesteps. On skewed occupational prompts, SGS reaches high intra-image racial and gender diversity while keeping aggregate balance competitive and image quality near baseline, a result we validate on two architecturally distinct diffusion transformers, FLUX.1-Dev and SD3.5-Large. Codebase and model weights are available at https://github.com/cvims/spatially-gated-steering