Few-step generative models aligned using direct preference optimization
FestDPO: Few-step Generator Alignment with Direct Preference Optimization
Machine Learning
Summary
Some computer programs can quickly create things like images or protein shapes, but these creations might not always meet what we want. The authors worked on a way to improve these programs by teaching them with human preferences directly, without needing to guess scores in between. They developed a new method called FestDPO that can handle these quick programs even though they don’t easily show how likely their outputs are. Tests showed FestDPO can better align creations with human preferences in pictures and proteins.
What this means in practice
- •For graphic designers: Improve text-to-image generation tools by directly tuning models to human preferences without separate reward models.$Commercial implications: Enables development of image generation products that produce outputs better aligned with user preference rankings.
- •For protein engineers: Enhance protein backbone design by guiding generative models with preference feedback to yield more structurally desirable outputs.
Authors
Jaewoo Lee, Kyuil Sim, Hyeongyu Kang, Kanghoon Lee, Woocheol Shin, Jinkyoo Park
Abstract
Few-step generative models can generate high-fidelity samples within a few function evaluations. Despite this efficiency, generated samples may not exhibit desirable properties. When these properties are difficult to encode as an explicit reward function, direct preference optimization (DPO) can align generative models using pairwise preference feedback without training a separate reward model. However, extending DPO to few-step generative models is challenging because few-step generative models are generally implicit, making the likelihood evaluation required by DPO intractable. To address this challenge, we introduce Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples. By exploiting the fast sampling capabilities of few-step generative models, our approach makes sample-based approximation of DPO loss computationally feasible. Furthermore, the sample-based formulation makes FestDPO agnostic to the model family and sampling procedure. Our toy experiment demonstrates that FestDPO matches the reward-tilted target distribution across four few-step generators. For real-world tasks, we evaluate FestDPO in two domains: text-to-image generation and protein backbone generation. In text-to-image generation, FestDPO outperforms preference optimization baselines in both win rates against the base models and human evaluation scores. In protein backbone generation, it achieves a higher $β$-sheet fraction and better structural designability than the baselines.