Server method trims unlearning cost in shared language model training
Tracing Knowledge Lineage for Federated Unlearning under Mutual Distillation
Networking and Internet ArchitectureDistributed, Parallel, and Cluster Computing
Summary
When multiple devices learn a language model together, removing data from one device can be tough because its influence spreads everywhere. The paper's authors designed a method called HERD that tracks how knowledge moves between devices and picks the smallest set that must be fixed to erase the unwanted data effectively. They show when and why fixing only some devices works and prove their method reduces leftover unwanted data more cheaply than older ways.
What this means in practice
- •For edge server operators: Reduce computation and communication cost when removing outdated or incorrect data from collaboratively trained language models at the edge.
- •For mobile device manufacturers: Improve data privacy controls on devices by efficiently erasing user data from shared language models under mutual learning setups.
Authors
Zihao Ding, Jun Huang, Guodong Wang
Abstract
An edge server and nearby devices can train language models of different sizes together by mutual distillation. Federated unlearning (FU) and unlearning for large language models (LLMs) execute a removal request on one model. Under mutual distillation, however, the effect of the removed data has been copied into many students, and their uploads teach it back to the corrected teacher. We propose Holder-level Erasure via Replication-weighted Delta-closure (HERD), which decides which holders must be corrected. HERD records a knowledge lineage graph from the transfers that the server already performs and weights each edge by its transfer type. It computes how many times the residual at each holder will be copied and selects the cheapest set of holders whose correction bounds the cumulative residual after the request. A transfer guard and a rejoin rule cover the holders left uncorrected or offline. We prove that partial correction suffices only when the copy number of the lineage graph is below one, that averaging alone never reaches one, and that the correction set remains valid under link outages. On TOFU, HERD leaves 18% to 52% less cumulative residual than the strongest baseline at 6% to 12% of its correction cost.