Contrastive sparse autoencoder improves selective forgetting in ai models
Neuralyzing the Trace: Selective Representation-Level Unlearning with Contrastive Sparse Autoencoders
Artificial Intelligence
Summary
Machine unlearning is about teaching AI models to forget specific information without harming their overall abilities. The authors find that usual methods tend to miss subtle, low-energy details related to what should be forgotten. They introduce a new tool called SCALPEL, which uses a contrastive sparse autoencoder to better identify and remove those specific details. Their experiments show SCALPEL works better than common existing techniques on several AI models.
What this means in practice
- •For ai engineers: Improve AI model compliance by selectively removing user-specific data while maintaining other model functions.
- •For data privacy officers: Support GDPR privacy requests with more precise removal of personal data from trained models.
Authors
Itai Zehavi, Fanny Jourdan, Ulrich Aivodji
Abstract
Machine unlearning aims to remove targeted information while preserving a model's other abilities. In realistic settings, such as privacy requests under the EU GDPR, the target may be narrow, for example information associated with a single person. Behavioral forgetting alone may be insufficient, motivating interventions directly on internal representations. However, standard mechanistic-interpretability extractors are poorly selective for such targets. We identify an energy bias in reconstruction-based extraction, which favors dominant background structure over low-energy target-specific components. We introduce SCALPEL, a contrastive sparse autoencoder designed to learn more selective forget features. We show theoretically that contrastive training promotes target-selective features and that our selection score controls expected background knowledge perturbation. We validate SCALPEL experimentally on TOFU across Qwen, Llama, and Gemma, where it substantially improves over NMF and standard SAE interventions and is competitive with Gradient Difference and RMU, bridging mechanistic interpretability and fine-grained unlearning.