Text to image models improved by geometry aware preference tuning
Geometry-Aware Preference Optimization for Text-to-Image Diffusion Models
Computer Vision and Pattern Recognition
Summary
Text-to-image AI models often learn to create images that match human preferences by optimizing what they produce. The authors found that current methods do not consider the natural structure of image data, which lies near a simpler shape in a very complex space. Because of this, the models sometimes lose image quality and variety when trained for preferences. They propose a new method that respects the geometry of the image space, guiding the model to improve images without breaking the natural structure. This leads to better images and more diversity with less training.
What this means in practice
- •For ai developers: Train text-to-image models that produce higher quality and more diverse images aligned with user preferences using fewer training steps.
- •For digital artists and content creators: Generate images from text prompts with improved semantic consistency and natural variation by using models optimized with geometry-aware preference methods.
Authors
Lei Wang, Zhen Wang, Yuexiang Xie, Yaliang Li
Abstract
Preference alignment has become a standard practice for text-to-image diffusion models. Direct Preference Optimization (DPO) simplifies this process by eliminating explicit reward modeling. Its diffusion variant, Diffusion-DPO, has become a widely adopted baseline. Diffusion-DPO essentially encourages the likelihood of preferred samples while suppressing dispreferred ones. In this paper, we revisit DPO-style alignment methods for diffusion models from the perspective of the manifold hypothesis. Under this view, natural images concentrate near a low-dimensional manifold embedded in the high-dimensional ambient space, whereas DPO directly optimizes preference distributions in the full space without accounting for this geometric structure. This creates a mismatch in the optimization dynamics: it suppresses geometry-preserving tangential updates, while insufficiently restricting hazardous normal-direction updates. This mismatch gradually degrades image quality and diversity. To address this issue, we propose Anisotropic Geometry-Aware Preference Optimization (APO), which replaces the uniform Euclidean treatment of prediction errors with a geometry-aware anisotropic metric derived from the reference model. Concretely, APO adaptively strengthens regularization in directions where the reference denoising function is highly sensitive, while relaxing constraints in directions that permit safe semantic adjustment. This recalibrates preference optimization according to the local manifold geometry, and maintains the original manifold structure. Experiments show that APO achieves strong performance and an average win rate exceeding 60\% against various existing alignment methods across diverse benchmarks. It requires significantly fewer training steps than prior methods, and preserves generation diversity throughout training.