Continual personalization of image models with fixed size adapters
CLASP: Continual Low-rank Adapters for Spatially Placed Concepts from One Hypernetwork
Computer Vision and Pattern Recognition
Summary
Personalizing image generation models with new concepts usually means the models forget old ideas or need more space to remember everything. The authors propose a new way to keep teaching a model new things without forgetting the old ones and without making the model bigger. They do this by using a special small network that creates customized tweaks for each new concept while keeping everything else unchanged. This also lets users place the new concepts exactly where they want in the image.
What this means in practice
- •For graphic designers: Add new visual concepts into image generation tools without losing previous custom content or increasing model size$Commercial implications: Enables scalable personalized image generation software for designers seeking continuous creative customization without hardware upgrades
- •For game developers: Embed evolving personalized character or item appearances in games using fixed-size model adaptations that maintain past designs
Authors
Wojciech Gromski, Patryk Krukowski, Jan Miksa, Maciej Zieba, Przemysław Spurek
Abstract
Continual personalization of text-to-image diffusion models requires sequentially acquiring new concepts while retaining previously learned ones. However, existing methods either suffer from catastrophic forgetting or rely on storing additional concept-specific parameters and spatial components, causing their parameter footprint to grow with the concept stream. This limits their ability to scale to long sequences of personalization tasks. We propose a rehearsal-free approach that uses a single fixed-size hypernetwork to continually personalize a frozen diffusion model. Instead of expanding the model as new concepts are acquired, the hypernetwork dynamically produces the concept-specific adaptations required for personalization while preserving previously learned concepts. Our framework further integrates spatial control into the personalization process, allowing users to specify where a personalized concept should appear without introducing additional per-concept components. This formulation enables continual personalization with a parameter footprint that remains independent of the number of learned concepts, aside from compact concept representations. Experiments demonstrate strong retention of previously learned concepts and reliable spatial grounding, matching or improving upon existing methods while scaling effectively to long streams of personalization tasks.