AdvFidelity unifies preference alignment and fast image generation in reinforcement learning

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

Computer Vision and Pattern Recognition

Summary

Creating images quickly while matching human preferences is hard because training often happens in multiple steps that can reduce quality. The authors show that reinforcement learning can improve the big picture but miss small flaws. They introduce AdvFidelity, which trains a system to spot fine details and low-level problems by comparing noisy versions of images, guiding the generator to fix those issues in fewer steps. This method works within a single training phase and creates high-quality images as effectively as longer, complex approaches.

What this means in practice

  • For digital artists: Generate images aligned with human preferences efficiently using fewer iterative steps in creative tools.
  • For machine learning engineers: Integrate advanced fidelity supervision into reinforcement learning pipelines for high-quality, fast image synthesis without multi-stage distillation.

Authors

Sicong Kang, Fangao Zeng, Yuhan Li, Hao Zhou, Mengfei Xu, Wenxiang Shang, Wei Li, Bingbing Ni, Pipei Huang

Abstract

Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillation is often multi-stage and may erode RL gains We observe that few-step RL progressively improves structural coherence, even with specialized rewards, but leaves low-level artifacts that these rewards may fail to penalize. To address this gap, we propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into RL through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without distribution matching distillation (DMD) or other explicit distillation objectives. We train a discriminator to rank high-quality references (positives) above content-matched reconstructions (negatives) produced by the current generator. Reconstructing moderately noised references largely preserves style and scene layout while exposing low-level degradations. This encourages the discriminator to assess artifacts rather than content differences. With negatives reconstructed online, alternating updates adapt fidelity supervision to the generator's evolving failure modes. References can come from existing target-domain data or full-step model samples. Built on LeapAlign, AdvFidelity backpropagates gradients from both rewards through the few-step sampling trajectories used at inference. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality and remains highly competitive with other evaluated full-step RL methods, using only 4 or 8 steps without CFG.