Federated large language models made more secure against faulty updates
Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing
Machine LearningCryptography and Security
Summary
Federated large language models let many users train a shared AI without sharing their private data. However, some users may send bad updates to the AI that can confuse or corrupt the model's knowledge. The authors propose a new technique that finds and blocks these harmful updates by looking at how updates relate to each other across multiple users. Their method helps keep the shared model accurate and consistent while still protecting everyone's data privacy.
What this means in practice
- •For cloud service providers: Improve defenses in federated AI platforms to maintain model quality despite malicious participant updates in decentralized training.
- •For financial fraud detection teams: Enhance collaborative model training among distributed agents while ensuring corrupted or misleading updates do not degrade detection accuracy.
Authors
Kai Li, Hanlin Cai, Zihao Liu, Yichen Li, Jose Alcaraz Calero, Wei Ni, Falko Dressler, Özgür B. Akan
Abstract
Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder (HMP-GAE) framework, where low-rank model updates are represented as nodes in a hypergraph and high-order semantic relationships among multiple agents are exploited through iterative message passing. By leveraging historical checkpoints and cross-agent relational reasoning, HMP-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training. HMP-GAE is implemented using Qwen-2.5 and Llama-3.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo! Answers datasets. Compared to existing defenses, HMP-GAE improves global accuracy by $1.21$\%, reduces final-model classification semantic entropy by $44.08$\%, and lowers perplexity by $13.34$\%. HMP-GAE outperforms the GAE-based defense by improving the accuracy $1.39$\%, reducing the semantic entropy by $54.5$\%, and lowering perplexity by $35.5$\%.