What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation
2026-08-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors point out that popular methods for evaluating generative models, like FID and KID, only use simple statistics and miss important differences in how data varies. They show that these metrics can even rate bad fake images better than real ones. To fix this, the authors created a new method called ZID that provides a more detailed diagnosis by combining multiple tests sensitive to different aspects of the data. ZID not only ranks how different two data sets are but also tells if the difference is due to less or more variety in the generated samples. Their tests show that ZID catches subtle problems that FID misses.
Generative modelsFréchet Inception Distance (FID)Kernel Inception Distance (KID)Distributional differencesMode collapseStatistical momentsPermutation testDispersionRank graphGaussian kernels
Authors
Hao Chen
Abstract
Generative models are commonly ranked by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can miss distributional differences, and a reported scalar gap alone is not a calibrated test against sampling variation. FID's moment restriction has concrete consequences: on ImageNet, visually unrecognizable images optimized only to match the reference Inception mean and covariance obtain FID $24.7$ versus $58.6$ for held-out real images (lower is better). Moreover, FID and KID are scalar discrepancies that are unchanged when the two samples are exchanged and therefore do not encode the direction of a dispersion change: under-dispersion, as can occur in mode collapse, versus over-dispersion. We introduce \textbf{ZID} (\emph{Z-resolved Integrated Diagnostic}), which combines six standardized location- and dispersion-sensitive arms from a rank graph (RISE) and Gaussian kernels (GPK at two bandwidths). Rather than asking one scalar to serve incompatible roles, ZID reports three linked outputs: an index for ranking departure magnitude, a permutation $p$-value for testing distributional equality, and a signed dispersion readout for diagnosis. In controlled experiments, ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps, including cases in which FID is flat or reversed. On DiT-XL/2 and SiT-XL/2 guidance sweeps, ZID detects departure from real data, and its signed readout labels the high-guidance diversity collapse as under-dispersion.