DiffusionART improves efficiency and stability in tuned diffusion models
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Machine Learning
Summary
Fine-tuning diffusion models with reinforcement learning is often slow or unstable. The authors found hidden forces in current training methods that cause instability and inefficiency. They created DiffusionART, a new approach that combines rewards across prompts, removes these forces, and stabilizes training without tricky tuning. This method works faster, uses fewer samples, and performs reliably on both images and videos.
What this means in practice
- •For image generation engineers: Train diffusion-based image generators more efficiently with stable reward fine-tuning requiring fewer resources.
- •For video synthesis developers: Improve training speed and reliability for video generation models using combined visual and motion quality feedback.
Authors
Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You
Abstract
Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.