Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning
2026-06-22 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how to keep large language models updated when many devices work together but can join or leave at any time. They propose a new method using something called orthogonal LoRA to let the system add or remove a device's learning without needing to remember all the past changes. This approach also uses a smart policy to decide the best way to fix the model depending on what kind of problem happens when devices change. Their experiments show their method helps the model adjust quickly and correctly when devices come or go.
large language modelsdecentralized federated learningorthogonal LoRAparameter-efficient tuningpeer-to-peer networksmodel fine-tuningdynamic networksresource allocationpost-training correctiontopology adjustment
Authors
Nuocheng Yang, Yechen He, Sihua Wang, Zihan Chen, Tony Q. S. Quek, Changchuan Yin
Abstract
As large language models (LLMs) are increasingly deployed at the network edge to provide pervasive generative AI services, decentralized federated learning (DFL) provides a vital mechanism for privacy-preserving, domain-specific fine-tuning through peer-to-peer exchanges of parameter-efficient updates. However, the dynamic nature of practical decentralized edge networks, where devices may dynamically join or leave the collaborative training process, requires the system to continuously adapt to new data while selectively removing prior contributions. This correction process remains a significant bottleneck, as individual device updates become deeply entangled within the global fine-tuned parameters. To address this challenge, we propose a priority-aware learning-unlearning correction framework based on orthogonal LoRA that can enhance the knowledge evaluation through topology adjustment. Specifically, we first design an orthogonal LoRA mechanism that yields post-training contribution coordinates, enabling history-free projection addition and deletion in response to membership changes. We then analyze the correction bottleneck and develop a priority-aware policy that selects among topology refinement, local correction, proximal damping, and synchronization scheduling according to the dominant residual term. A resource allocation algorithm is further developed to allocate limited communication across layer groups, prioritizing the primary bottlenecks within per-round wireless constraints. Experiments demonstrate that the proposed framework achieves robust post-event correction for both device join and leave events and validate that different residual regimes necessitate distinct correction actions.