Summary
Graphs are useful for representing all kinds of connected information, but teaching computers to understand them can be tricky, especially across different kinds of graphs. The authors created a new way to train a graph model using synthetic graphs made with a special kind of logic called structural causal models, which helps the model learn how nodes and connections influence each other. Their model, called GSCM-PFN, can predict well on many real-world graph problems despite not being trained on any real graph data beforehand. This approach helps overcome limits of older models that either needed lots of real data or had to rely on text-based information.
What this means in practice
- •For machine learning engineers: Build graph models that generalize across domains using pre-training solely on synthetic causal graphs.
- •For data platform developers: Create graph data synthetic generators that incorporate causal relationships for diverse downstream tasks.
Authors
Ziwei Zhang, Tianyin Liao, Chunyu Hu, Ge Lan, Nuo Xu, Yi Huang, Qingyun Sun, Xingxuan Zhang, Peng Cui, Jianxin Li
Abstract
Graph Foundation Models (GFMs) have recently emerged as a promising paradigm for learning transferable representations across different graphs and domains. However, existing GFMs face notable limitations, including limited pre-training scale or reliance on aligning with textual information in Large Language Models (LLMs). To overcome these limitations, recent studies have drawn inspiration from prior-data fitted networks (PFNs), which learn from large synthetic datasets and support multi-domain prediction backed by causal learning. However, existing graph PFNs encounter severe difficulties in modeling causal relationships within and across node and edge sets, generating diverse graph structures, and designing suitable architectures that can learn from complex synthetic datasets. To address these challenges, we propose **GSCM-PFN** (Graph SCM-based Prior-Data Fitted Network), a graph PFN pre-trained on synthetic data generated from an SCM-based prior that jointly models nodes and graph topology. Specifically, we (1) design a joint feature-edge-label structural causal model (SCM) to guide synthetic graph generation and capture causal relationships within and across node and edge sets; (2) introduce prior-guided node-edge dependency modeling to instantiate diverse graph generation strategies, and prove that our SCM-based generation prior is more expressive than existing priors; and (3) propose a feature-permutation-invariant dual-path graph network with theoretical guarantees for learning from diverse synthetic graphs. Experiments on 23 real-world benchmarks show that GSCM-PFN outperforms both conventional GFMs and PFN-based GFMs, without utilizing real-world graph data for pre-training. Code: https://anonymous.4open.science/r/GSCM-PFN-A3A3/