Federated learning method improves knowledge removal in large language models
FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning
Machine LearningArtificial IntelligenceComputation and Language
Summary
When multiple users train language models together without sharing raw data, removing specific unwanted knowledge while keeping useful information is hard. The authors identified that simply combining local updates can cause the model to become unstable or forget too much useful content. They propose FIRM, a technique that uses special statistics from each user's training to carefully combine their updates, keeping the model stable and preserving knowledge meant to be retained. Experiments show FIRM balances forgetting unwanted knowledge and keeping useful information better than previous approaches.
What this means in practice
- •For federated learning engineers: Integrate FIRM to safely remove targeted knowledge from federated models without harming retained information across clients.
- •For privacy compliance teams: Support legal and ethical requirements by selectively unlearning sensitive data from federated LLM deployments using FIRM.
Authors
Reeyansh Shah, Maram Hasan, Subhasis Chaudhuri, Biplab Banerjee
Abstract
Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.