FedRAPP improves federated adaptation for vision language models
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Computer Vision and Pattern Recognition
Summary
Federated learning allows multiple devices to train AI models together without sharing private data. The authors found that when devices get new, unlabeled images that differ from what they trained on, existing methods struggle to combine their AI updates effectively. They propose FedRAPP, which smartly weighs updates from trusted devices and keeps some local learning to handle differences better. Tests on many image datasets show that FedRAPP works better at adapting AI models under varied conditions while still being efficient.
What this means in practice
- •For mobile app developers: Update vision-language models on users' devices collaboratively without sharing data, improving performance under diverse conditions with unlabeled images.
- •For cloud ai platform teams: Enhance federated model tuning for applications receiving unlabeled data streams from distributed clients with different data patterns.
Authors
Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali, AKM Mahbubur Rahman
Abstract
Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data. However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions. At test time, source-free federated prompt tuning remains largely unexplored. Clients may receive unlabeled target samples with heterogeneous, often non-IID distributions that differ substantially across clients and from the source training distribution, causing locally adapted prompts to drift in different directions and reducing the effectiveness of federated aggregation. In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data. Our study reveals that conventional federated aggregation methods struggle under heterogeneous client distributions, as locally adapted prompts become increasingly specialized and difficult to combine effectively. To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation. FedRAPP selectively incorporates information from reliable and compatible clients while preserving local specialization, enabling more robust adaptation under distribution shift. Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.