Restoring variety in fast image and video generation models
Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Computer Vision and Pattern Recognition
Summary
Generating images and videos quickly often involves making simpler versions of big models, but this can cause the models to produce very similar results every time. The authors found that while these simpler models keep overall activity similar to the original, this activity is focused too narrowly, losing diversity. They created two methods to fix this by restoring the lost variation in the model’s internal signals, leading to more varied outputs without losing quality. Their work applies to both pictures and videos and was confirmed by user feedback.
What this means in practice
- •For visual content creators: Produce diverse images and videos quickly by using distilled models enhanced with variance restoration techniques to prevent repetitive outputs.
- •For mobile game developers: Integrate fast and diverse image generation in games by using distilled diffusion models improved to maintain output variety.
Authors
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
Abstract
Distilling diffusion models into few-step student models substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study.