IREU: Identity-Related Encoder-Only Unlearning for Customized Portrait Generation
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to make portrait generation models forget or 'unlearn' certain people to protect privacy, since these models can create fake images of anyone. They first tried a simple method that reduced how much the model recognized target identities but found it also hurt image quality for others. To fix this, they developed a new method called IREU that carefully tweaks only the parts of the model related to the target identities, improving forgetting while keeping good image quality for others. Their approach also works well with different generators without extra adjustments, making it practical to use.
Customized Portrait GenerationIdentity UnlearningImage EncoderFeature SpacePrivacy ProtectionImage FidelityOffline Feature IdentificationModel GeneralizationFeature Perturbation
Authors
Chaoyi Shi, Shanshan Zhang, Jian Yang
Abstract
Customized Portrait Generation (CPG) technologies have been widely used to generate high-fidelity person images given an input image indicating the identity and a text prompt indicating the required edits. Yet these methods pose significant privacy risks by spreading fake visual information. Against such risks, each public generator should be able to suppress its generation ability for a particular person when requested. Therefore, in this work we investigate the identity unlearning problem for CPG. Since there are no previous methods in this field, we propose a simple baseline that updates the image encoder by minimizing identity similarity between generated and input images for target identities to be unlearned, while maximizing it for identities to be retained. However, we find such a global perturbation in the feature space harms the fidelity of generated images for other identities to be retained. To solve this problem, we propose a novel method IREU, which first locates identity-related features in an offline manner and then only performs feature perturbations on them. The experimental results show that our proposed method IREU achieves better identity unlearning performance for target identities to be unlearned, and also keeps high fidelity for other identities to be retained. In addition, our unlearned image encoder is generalizable across different generators with the same encoder without fine-tuning, which is friendly for deployment in practice.