Pretrained graph models improve relational data prediction without fine-tuning
Path to Prior-Data Fitted Relational Models via Learned Latent Multi-table Representations
Machine LearningArtificial Intelligence
Summary
Understanding data spread across multiple tables can be tricky, especially when the data is connected like a graph. The authors study a kind of machine learning model that mimics how information passes between connected nodes and pretrained it on lots of made-up databases. They found that this pretraining helps the model understand real-world data better, even without extra tweaking. Their approach uses synthetic data in a step-by-step way that leads to stronger predictions across many tasks.
What this means in practice
- •For enterprise data engineers: Build relational models that generalize well to new structured databases without needing to retrain or fine-tune on each new dataset.
- •For software developers: Integrate pretrained graph-based encoders to extract useful features from relational data for various classification problems directly through latent representations.
Authors
Mohammad Sadeq Abolhasani, Viswanath Ganapathy
Abstract
Prior-data fitted networks (PFNs) show that pretraining on large populations of synthetic tasks can enable generalization to unseen datasets with little or no downstream fine-tuning. This paradigm is effective for transformer-based tabular and relational foundation models such as RDB-PFN, but remains largely unexplored for graph-based relational models built on message passing. We pretrain the graph-based relational foundation model Griffin on large collections of synthetic relational databases and ask whether curriculum learning, which benefits transformer-based relational learners, also improves message-passing representations. Rather than fine-tune on downstream tasks, we use the pretrained encoder as a general-purpose relational representation learner and pass its latent representations to a frozen in-context predictor (\ticl{}); no component is trained on the target task. On 16 real-world relational classification tasks, evaluated over five seeds against raw-feature and untrained-encoder baselines, a purely synthetic curriculum that begins with schema-guided data and continues with schema-agnostic data gives the strongest frozen in-context performance\rev{, outperforming real-data-only pretraining and every unordered single-stage mixture of the corpora; when appended after real-data pretraining, it yields the best encoders overall}. These results provide initial evidence that PFN-style synthetic pretraining transfers to graph-based relational foundation models.