Multimodal Federated Learning under Dual-Axis Modality Missingness
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors present Flux, a system for federated learning where different users have different types of health data, and sometimes only parts of that data are available for each test. Flux uses a smart way to estimate how confident it should be about each type of data, adjusting predictions based on how complete and reliable the information is. It separates confidence adjustment from the shared learning phase to keep the model stable across users. Their tests on several health-related datasets showed Flux performed better than existing methods and maintained reliable predictions even with missing or corrupted data.
multimodal federated learningmodality missingnessconfidence temperingfederated learningunimodal supervisiongradient decouplingpredictive calibrationhealth sensingshared model optimization
Authors
Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin, Yunqi Guo, Xiaomin Ouyang, Guoliang Xing, Steven Euijong Whang, Sung-Ju Lee
Abstract
Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.