UltraDiff improves high resolution image generation efficiency and quality

UltraDiff: Transferring High-Fidelity Priors to Compressed Latent Spaces for High-Resolution Image Generation

Computer Vision and Pattern Recognition

Summary

Generating very detailed images using computer models often requires a trade-off between speed and image quality. The authors created UltraDiff, a new method that combines detailed and compressed image information to produce high-quality images much faster than before. Their approach uses a special teacher-student process to teach the model how to avoid losing important details during compression. As a result, UltraDiff can generate large, high-resolution images quickly without sacrificing quality.

What this means in practice

  • For digital artists: Create high-quality 2K to 4K images faster using efficient latent diffusion techniques balancing speed and fidelity.
  • For game developers: Generate detailed textures and backgrounds at scale with significantly reduced compute time using UltraDiff’s compressed latent space approach.

Authors

Jingjing Ren, Haitian Zheng, Haoyu Chen, Connelly Barnes, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Lei Zhu

Abstract

High-resolution synthesis with Latent Diffusion Models (LDMs) faces a fundamental trade-off between computational cost and generative fidelity. This trade off is largely governed by the latent compression factor f (e.g. f8 denotes f=8 and f32 f=32). While aggressive compression (f32) significantly improves efficiency, it suffers from severe information loss and ambiguity compared to standard f8 models. We propose UltraDiff, a framework that reformulates this challenge as joint latent diffusion learning followed by marginal score distillation. We introduce MixTeacher, a dual branch DiT that models the joint f8-f32 distribution, utilizing shared attention to resolve the ambiguity of the compressed space with high fidelity f8 priors. Crucially, we propose vector field distillation in the native f32 space. Theoretically, this compels the student to learn the conditional expectation of the teacher trajectory, effectively filtering out compression artifacts. Across the evaluated 1K to 4K settings, UltraDiff approaches f8 level quality while preserving f32 efficiency. With 20 sampling steps on a single A100 GPU, it generates a 2K image in 6.2 seconds over 40 times faster than our f8 baseline.