Summary
People often think that if a computer model can recreate images accurately, it will also generate new images well, but this isn’t always true. The authors found this is because the model sees different kinds of data when reconstructing versus generating images. They created a method called generation-aware reconstruction (GAR) that gradually shifts the data from reconstruction mode to generation mode, allowing them to better understand and measure this difference. Their new measurement, GAR-FID, predicts how well a model will generate images more accurately. This approach also helps improve the models by adapting the decoder to be better with the types of data it encounters when generating images.
What this means in practice
- •For machine learning engineers: Improve image generation quality by adapting decoders using intermediate latents characterized along the GAR trajectory.
- •For computer vision developers: Evaluate generative models more reliably using the GAR-FID metric that better correlates with real generation performance.
Authors
Xianghong Fang, Wenjie Shu, Tongda Xu, Wenlong Mou, Dehan Kong, Tim G. J. Rudner
Abstract
In latent generative models, reconstruction quality is often assumed to correlate with generative performance. However, reconstruction FID (rFID) can exhibit weak or even negative correlation with generation FID (gFID). We attribute this discrepancy to a latent distribution mismatch: reconstruction evaluates the decoder on encoder-induced latents, whereas generation uses the same decoder on latents produced by the generative model. To characterize this shift, we introduce generation-aware reconstruction (GAR), which constructs a continuous trajectory from standard reconstruction toward generation by perturbing encoder latents with noise and denoising them through the generative model before decoding. GAR probes the decoder behavior along this trajectory, making the transition from encoder to generation-time latent distributions observable and diagnosable. The resulting trajectory-based diagnostic, GAR-FID, exhibits strong empirical correlation with gFID across diverse tokenizers and scales. Importantly, intermediate GAR latents become more generation-aware while preserving correspondence with their source images, thereby retaining paired supervision that is absent for fully generated latents. This correspondence enables decoder adaptation on intermediate GAR latents, consistently improving generative quality across model scales. Overall, latent distribution mismatch provides a useful perspective for evaluating and improving latent generative models.