Fast slow learning improves graph models with diverse domains
More Domains, No Gains? FLO: Fast-Slow Reciprocal Learning for Multi-Domain Graph Pre-training
Machine Learning
Summary
Learning from many different types of graph data usually helps models get smarter, but current methods don't always gain from using more varied data. The authors found two main problems: models learn relationships mostly through node connections, missing other useful info, and mixing knowledge from different domains too early causes interference. They created a new method called FLO that combines a fast learning part adapting to each domain with a slow part combining knowledge across domains, plus a way to capture attributes without relying on connections. FLO performs better than previous methods on many datasets.
What this means in practice
- •For graph machine learning engineers: Train graph models that better learn from diverse multi-domain graph data, improving transfer to new tasks and domains.
- •For data integration teams: Improve integration of knowledge from multiple graph data sources without interference during model training.
Authors
Yupeng Tong, Lianze Shan, Jitao Zhao, Di Jin, Dongxiao He
Abstract
Inspired by the great success of foundation models in the development of artificial general intelligence, Graph Foundation Models (GFMs) have recently attracted significant attention and have the potential to advance graph intelligence. Exhibiting such intelligence requires effectively learning knowledge from large-scale, multi-domain graph data during pre-training. Increasing domain diversity during pre-training is expected to yield richer and more generalizable knowledge. Most existing methods pursue this goal by designing domain alignment strategies that help a single graph model learn from multiple domains. However, we observe that existing methods struggle to benefit from increasing domain diversity. Through empirical and theoretical analysis, we reveal two key limitations. Firstly, pre-training in most GFMs is confined to a single message-passing paradigm. Since propagation flattens relations among neighboring nodes, the graph model can hardly perceive the knowledge beyond propagation, namely the attribute-predictable component of these relations. Secondly, domain alignment strategies often rely on randomly initialized parameters that are jointly updated with the graph model using immediate loss feedback, leaving early knowledge acquisition vulnerable to cross-domain interference. To address these issues, we propose FLO, a Fast--sLow reciprOcal approach for multi-domain graph pre-training. Specifically, a propagation-free attribute branch predicts the graph model's relations without message passing and returns these predictions to refine the model. We further stabilize knowledge acquisition by pairing a fast process that adapts to the current domain with a slow process that integrates knowledge across domains. On 12 datasets, FLO benefits from increasing domain diversity and achieves the best average performance in cross-domain and cross-task transfer.