Client selection improves federated multimodal model fine-tuning under network limits
CARE-FL: Federated Fine-Tuning of Multimodal LLMs in Resource-Constrained Edge Networks
Distributed, Parallel, and Cluster Computing
Summary
Training large language AI models that understand images and text together is hard when data is spread across different devices that have different connection speeds and data types. The paper offers a new way to pick which devices should help train the model so the process is faster and uses less network data. Their method learns over time which devices to choose, balancing speed and bandwidth limits. Tests show it gives better and more stable AI performance on disaster response tasks than older methods.
What this means in practice
- •For edge network operators: Efficiently fine-tune AI models across wireless devices while respecting network speed and data limits in disaster response scenarios.
- •For mobile ai engineers: Develop AI features that adapt better to heterogeneous multimodal data from smartphones with varying connectivity and computing power.
Authors
Tianqi Zong, Baosheng Li, Nuocheng Yang, Binbin Chen, Marie Siew
Abstract
Domain-specific adaptation of Multimodal Large Language Models (MLLMs) in privacy-sensitive edge environments requires federated fine-tuning, yet client selection remains challenging due to heterogeneous multimodal data and modality availability, dynamic system conditions, and evolving client utility during training. We propose Constraint-Aware Reinforcement Learning for Efficient Federated Fine-Tuning (CARE-FL), a constrained online client-selection framework for federated MLLM Low-Rank Adaptation (LoRA) fine-tuning. CARE-FL formulates client participation as a sequential decision process and employs Q-value decomposition to address the large combinatorial joint action space through scalable client-level long-term utility estimation. These utilities are learned online using a modified Double Deep Q-Learning (DDQL) scheme, while constraint-aware $ε$-greedy execution selects feasible client subsets under per-round latency and bandwidth constraints. Complementing client selection, an adaptive aggregation mechanism further controls potentially unstable global updates and places greater weight on the previous model when needed. Experiments on the CrisisMMD disaster-response task in emulated heterogeneous wireless edge environments show that CARE-FL achieves higher and more stable test accuracy than the compared client-selection baselines across different system settings, while satisfying the imposed latency and bandwidth constraints.