Previewdiff improves image and video generation by guiding diffusion early
PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Diffusion models create impressive images and videos but often mess up details like the number of objects or their positions. The authors introduce PreviewDiff, a method that checks the image as it’s being made and gets feedback using natural language to fix mistakes early. This way, it tries different ways to improve the picture before it’s finished. Their method works better than just picking the best final image from many tries and helps make images more faithful to the original instructions.
What this means in practice
- •For graphic designers: Create more accurate images from text prompts by improving early error detection during generation.
- •For video content creators: Generate videos with better adherence to detailed scene descriptions using guided searches in the creation process.
Authors
Vighnesh Subramaniam, Boris Katz, Brian Cheung, Chun-Liang Li, Tomas Pfister, Yale Song
Abstract
Diffusion models can produce striking images and videos, but they still struggle with the compositional details that make a generation faithful to a prompt, such as object counts, attribute binding, spatial relations, and temporally grounded actions. A common way to improve prompt satisfaction is to spend more compute at test time through Best-of-N sampling, but final-sample selection is fixed. Best-of-N can only choose among completed outputs and cannot repair a promising trajectory before it fails. We introduce PreviewDiff, a training-free test-time search method that turns diffusion sampling from scalar search into a multimodal critic-guided search over intermediate latents. At selected denoising checkpoints, PreviewDiff decodes a partial preview, asks a multimodal judge to score and critique it, and uses the resulting natural-language feedback to branch over semantic prompt edits and locally re-noised latent continuations. These branches are then scored and selectively rolled forward, allowing verifier compute to guide generation while the sample is still editable. Across image and video generation benchmarks, PreviewDiff consistently improves over budget-matched Best-of-N selection and strong scalar-search baselines. Ablations show that earlier interventions and increased search width provide the largest gains, while deeper search and additional semantic variants offer complementary improvements. PreviewDiff demonstrates that multimodal feedback is most useful not only as a final verifier, but as an active controller inside the denoising process.