Summary
Different users can have images and text that look quite different depending on their environment, which makes AI models less accurate when shared across devices. The authors show that just treating each user’s data style as a simple shift is not enough because how images and text link can vary by specific content categories. They propose a method called MOSAIC that focuses on correcting only the parts of these shifts that confuse the model when distinguishing correct text from wrong options. Their approach uses smart adjustments personalized per user and improves accuracy on several benchmark datasets involving multiple domain shifts.
What this means in practice
- •For mobile app developers: Improve personalized vision-language features on devices without sharing raw data by correcting harmful domain-specific shifts in user data.
- •For cloud service engineers: Enhance federated learning systems by integrating fine-grained class-domain corrections that preserve user privacy while boosting model accuracy.
Authors
Wentao Yue, Qingyu Mao, Tianyou Lai, Ahmed M. Abdelmoniem, Qilei Li
Abstract
Federated parameter-efficient fine-tuning enables distributed clients to adapt pretrained vision-language models without sharing raw data or updating the full backbone. Its effectiveness, however, is limited by domain heterogeneity across clients. Existing personalized methods separate globally shared knowledge from client-specific style, but they largely treat each domain as a class-agnostic transformation. We show that this abstraction is insufficient: the cross-domain displacement associated with a fixed domain varies across semantic classes, and only a subset of these class-domain residuals damages the image-text decision margin. We therefore propose Margin-Oriented Semantic-Appearance Interaction Correction (MOSAIC), which first constructs a decision-aware harmfulness score that measures whether a training-derived class-domain residual favors a competing text prototype over the true class. It then models fine-grained class-domain interactions with a low-rank residual adapter whose class factors and residual basis are globally shared while domain factors remain client-private. An image-conditioned gate further controls candidate-wise correction, and harmful-pair-aware reweighting prioritizes decision-relevant residuals during local optimization. Extensive experiments on Office31, OfficeHome, and DomainNet100 demonstrate that MOSAIC consistently improves macro-client top-1 accuracy across all evaluated domain-shift and joint domain-label-shift settings.