FlowCPO improves preference alignment in flow models using offline data
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Artificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Training AI models to agree with user preferences can be done by online learning from fresh samples or offline learning from fixed examples. The authors clarify how these methods are related and propose FlowCPO, which uses both good and bad examples together without needing new online samples. This approach performs better than existing methods on some standard tests of model quality but shows mixed results in new areas. FlowCPO also avoids some mathematical issues found in similar prior methods.
What this means in practice
- •For machine learning engineers: Improve AI model tuning using offline preference data without requiring time-consuming online samples.
- •For ai product teams: Use FlowCPO to enhance user-aligned AI features by leveraging both positive and negative preference samples effectively.
Authors
Yansen Han, Shengyi Liao, Peng Sun, Deyuan Liu, Yuanxing Zhang, Pengfei Wan, Tao Lin
Abstract
Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.