Diffusion models learn new concepts sequentially with less forgetting
Continual Concept Personalization in Diffusion Models via Residual-Space Adaptation
Computer Vision and Pattern RecognitionMachine Learning
Summary
When teaching AI to create pictures, it's hard to add new ideas without messing up the old ones. The authors developed a new way for these models to remember past ideas while learning new ones one after another. Their method finds the most important parts of the AI to adjust, letting it learn better with fewer changes. This helps the AI keep old concepts sharp while learning new ones efficiently.
What this means in practice
- •For graphic design teams: Update AI image tools with new user-specific styles or objects sequentially without retraining from scratch or losing prior customizations.
- •For mobile app developers: Integrate personalized image generation features that learn new user concepts over time efficiently, reducing model size and compute needs.
Authors
Yuzhi Liu, Enis Simsar, Giulia Lanzillotta, Thomas Hofmann
Abstract
Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that all concepts are available simultaneously. In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts. We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models. Our work achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72\%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.