PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning
2026-06-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a problem where different clients have graphs with either images or text but not both, which makes learning hard. They call this situation client-level modality deficiency and explain that missing information in graphs can cause errors to spread during learning. To fix this, they propose PRISM, a method that borrows missing information from other clients in a smart way based on the graph structure. Their experiments show PRISM helps clients with missing modalities perform better than existing methods.
federated learninggraph neural networksmultimodal dataimputationdecentralized learningmessage passingmodality deficiencycross-modal retrievalgraph propagationstructural meta-prompting
Authors
Zekai Chen, Miao Zhang, Jiayang Xing, Xunkai Li, Xun Wu, Rong-Hua Li, Guoren Wang
Abstract
Multimodal federated graph learning (MM-FGL) aims to collaboratively learn from decentralized graphs with text and images. However, real-world clients may not share a common modality basis: a visual-search client may contain image--interaction graphs but no seller descriptions, while a catalog client may provide text but no product images. We refer to this practical setting as client-level modality deficiency. Unlike random instance-wise missingness, a deficient client lacks the local semantic basis needed to reconstruct the absent modality. More importantly, in graph learning, incomplete representations initialize message passing, so imputation errors can be filtered, mixed, and amplified by the receiving topology. To address this gap, we propose \textbf{PRISM} (\textbf{P}roactive \textbf{R}etrieval and \textbf{I}mputation via \textbf{S}tructural \textbf{M}eta-prompting), a topology-aware federated cross-modal imputation framework. Rather than reconstructing the missing modality solely from local observations, PRISM recovers missing-modality semantics from the federation and introduces them into local graph propagation under topology-aware control. Experiments on six multimodal graph datasets across graph-centric and modality-centric tasks show that PRISM consistently improves modality-deficient clients, outperforming state-of-the-art baselines by \textbf{4.48}\% on average.