AI summaryⓘ
The authors introduce DistMoE, a way to train multimodal large language models across different clients without sharing all their data in one place. Each client gets its own specialized part of the model to learn from its own data, while a public part stays shared. To keep these client parts from drifting too far apart, the authors use a method that adjusts only small parts of the model using a mix of local and public data, avoiding complicated joint training. This lets the model flexibly combine knowledge from different domains during use without needing to know which domain each piece of data comes from. Their tests show that DistMoE works well for adapting to different visual-language tasks while keeping client data separate.
Multimodal Large Language ModelsMixture of ExpertsDistributed TrainingVisual-Language TasksFeedforward NetworkDomain AdaptationInstruction TuningRegularizationModular RoutingCross-Client Learning
Authors
Mainak Singha, Niccolò Biondi, Elisa Ricci, Subhankar Roy
Abstract
Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client-specific drift, we introduce a public-anchored expert composition stage that updates only routers and lightweight private projection adapters on a mix of local client data and public data, via an isotropic regularization loss, therefore making it cross-client rehearsal-free composition. During inference, DistMoE performs modular routing over public and private experts, enabling token-wise domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Codes are available at https://github.com/mainaksingha01/DistMoE.