Generative models boost sample diversity with Jacobian based perturbations
JIVE: Jacobian-Informed Volume Expansion for Diverse Generative Sampling
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Generative models that create images or other data sometimes produce very similar outputs, which limits diversity. The authors introduce JIVE, a method that adds carefully chosen tweaks to the generation process based on the mathematical properties of the model. This approach helps produce a wider variety of high-quality samples without extra training. It works efficiently by using classical numerical techniques, improving diversity in various generation tasks.
What this means in practice
- •For image generation teams: Increase visual variety in generated images for creative and design applications by using JIVE to enhance sample diversity without retraining.
- •For voice synthesis developers: Generate more diverse voice samples in text-to-speech systems using JIVE's perturbations to improve naturalness and variation.
Authors
Guangxun Zhang, Brian Cai, Boxuan Zhang, Chao Chen, Ruixiang Tang
Abstract
Generative models often suffer from mode collapse and limited sample diversity. While prior works attempt to mitigate this by jointly generating a batch of samples and repelling their trajectories, these heuristics do not explicitly maximize the diversity of the resulting endpoints. We introduce JIVE, a training-free framework that enhances generative diversity by injecting velocity perturbations aligned with the leading right singular subspace of the generator's endpoint Jacobian. By leveraging this local geometric structure, JIVE provably maximizes endpoint diversity while preserving sample quality. To maintain practical efficiency, we compute these perturbation directions via matrix-free iterations rooted in classical numerical linear algebra, requiring only a small computational overhead. Across different benchmarks, JIVE boosts both pixel and feature-level diversity in few-step and one-step generation.