Open framework unifies methods to improve forgetting in multimodal AI models
Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
Artificial Intelligence
Summary
As AI models that understand images and text get better, it’s important to remove sensitive or unwanted information from them without breaking how well they work. The authors created Open-MMUnlearning, a tool that brings together many ways to erase specific knowledge from such AI models and methods to test how well this works. They compare different ways to forget information and check how much utility stays in the model, as well as how reliable the evaluation methods are. Their findings help others compare and improve these unlearning techniques systematically.
What this means in practice
- •For ai development teams: Use a unified framework to test and improve methods that remove sensitive info from multimodal AI models while keeping their performance intact.
- •For data privacy engineers: Employ benchmarked unlearning techniques and evaluation tools to enhance privacy compliance for models handling image and text data together.
Authors
Junkai Chen, Yuhao He, Qianshan Wei, Junxiang You, Jingwen Shao, Junkai Lin, Zhongkai Yue, Xiaotian Ye, Zhengbo Jiao, Jiali Cheng, Zhijie Deng, Kening Zheng, Ruiqi Liu, Hadi Amiri, Yi Yu, Zhenan Sun, Qi Li, Ka-Ho Chow, Sijia Liu, Liang Wang, Jiaqi Li, Shu Wu
Abstract
As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pressing. Machine unlearning offers one approach to addressing these concerns by removing designated information from trained models while preserving unrelated capabilities. However, fragmented implementations and evaluation protocols, incomplete robustness testing, and limited understanding of metric reliability make progress in MLLM unlearning difficult to assess systematically. We introduce Open-MMUnlearning, an open-source, extensible framework that integrates target-model preparation, multimodal data processing, unlearning, and evaluation through shared interfaces and structured configurations. The framework supports five benchmarks spanning privacy, safety, and copyright, eight MLLMs from four model families, and twelve unlearning methods. Its evaluation suite jointly assesses forgetting effectiveness, retained utility, and robustness to model interventions, adversarial inputs, and membership inference attacks. Using a common evaluation protocol, we compare ten representative unlearning methods. In this comparison, GD and MIP-Editor tie for the highest overall score: GD achieves the highest Forget Quality, while MIP-Editor preserves more Model Utility. We further introduce a metric meta-evaluation protocol that tests faithfulness using models with controlled exposure to target knowledge and robustness under quantization and relearning. Among the thirteen evaluated metrics, BLEU achieves the highest aggregate reliability score. KS-Test attains the highest faithfulness AUC but performs less well on robustness. Together, the framework and these findings support reproducible comparison of MLLM unlearning methods and systematic assessment of evaluation reliability.