Image generation models improve with condition aware knowledge reuse
Knowledge Diversion: A Novel Paradigm for Pre-training Decomposable Image Generation Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Many current image generation models learn all their knowledge together, which makes it hard to reuse specific parts for new tasks. The authors introduce Knowledge Diversion, a method that separates knowledge into shared parts and condition-specific parts during training. This helps the model adapt better to different types of image generation tasks without needing to relearn everything. Their experiments showed this method matches or beats traditional models while being more efficient for adapting to new conditions.
What this means in practice
- •For software engineers: Build image generation systems that adapt efficiently to varying input types by selectively reusing pretrained knowledge units.
- •For game developers: Create generative content tools that quickly customize images under different conditions, reducing retraining time for diverse game assets.
Authors
Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, Xin Geng, Yong Rui
Abstract
Large-scale pre-trained models have become foundational to modern machine learning, yet conventional pre-training objectives primarily optimize source-domain performance without explicitly considering downstream requirements. Consequently, transferable and task-specific knowledge remains entangled within a shared parameter space, limiting selective knowledge reuse and efficient downstream adaptation. This limitation is particularly consequential for image-generation models, which must accommodate diverse conditions and domains. To address this challenge, we introduce Knowledge Diversion (KDiv), a condition-aware pre-training paradigm for learning inherently decomposable generative models. Before pre-training, KDiv factorizes randomly initialized weights into rank-one Knowledge Units, which are organized into learngenes and tailors. During pre-training, continuous condition-aware routing consolidates knowledge shared across conditions in learngenes while directing condition-dependent variations to tailors. This structured organization enables learngenes to capture transferable generative priors and tailors to develop condition-specific expertise, allowing the resulting model to be selectively recombined for diverse downstream conditions. Extensive experiments across class-, text-, and image-conditioned generation demonstrate that KDiv consistently matches or surpasses dense baselines at comparable pre-training cost and achieves the strongest average downstream performance with a trainable parameter budget comparable to low-rank adaptation methods.