Multilingual benchmark measures machine unlearning across languages
$μ^2$-Bench: A Multilingual Machine Unlearning Benchmark
Computation and Language
Summary
Some language models can remember and share unwanted or harmful information in many languages, even if trained on just a few. The authors created a new test called μ²-Bench to see if a model really forgets bad information in all languages after being told to unlearn it. This test checks many languages and tracks how the knowledge spreads and disappears. Their results suggest that effective forgetting needs special methods considering how languages connect. This can help make language models safer and respect privacy better worldwide.
What this means in practice
- •For language model developers: Evaluate how well models forget specific content across diverse languages with an end-to-end benchmark.
- •For privacy compliance teams: Assess if sensitive information is properly removed from multilingual models after unlearning steps.
Authors
Kyomin Hwang, Hyeonjin Kim, Hyunho Lee, Yearim Kim, Yeji Song, Nojun Kwak
Abstract
Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indirect cross-linguistic spread. Multilingual Machine Unlearning (MMU) aims to remove such information, yet its evaluation remains underexplored, leaving unclear whether unlearning truly eliminates target knowledge across all languages. To bridge this gap, we introduce $μ^2$-Bench, an MMU benchmark that simulates the full pipeline of memorization, unlearning, and evaluation across diverse languages. It 1) spans a broad set of languages, 2) evaluates on both training and hold-out languages, and 3) assesses knowledge as dispersed across multiple languages. We show that successful MMU requires methods that reflect multilingual characteristics, and conduct analysis to provide deeper insights into MMU.