Graph foundation model trained on web graph enables versatile tasks
Training Graph Foundation Models on The Web Graph
Artificial IntelligenceMachine Learning
Summary
Understanding and analyzing large networks like the web is difficult because each graph has different features and labels. The authors introduce Acacia, a new kind of graph model trained from scratch on the web graph that can handle many tasks like classifying nodes or predicting links without extra training. It works with various types of features and doesn’t depend on message-passing from large language models. This shows that graph models alone can develop advanced abilities similar to language models.
What this means in practice
- •For data scientists: Use a single model to perform multiple graph tasks on web-scale networks without retraining for new features or labels.
- •For social network engineers: Predict new connections and classify user groups in social graphs using a model trained entirely on web data without external language components.
Authors
Ryoma Sato
Abstract
We introduce Acacia, a graph foundation model, trained on the web graph. Acacia (i) supports arbitrary feature dimensionalities and semantics without additional training, (ii) supports a wide range of tasks, including node classification, link prediction, node clustering, and graph generation, without additional training, (iii) has in-context learning capabilities, and (iv) does not rely on pretrained LLMs. In particular, existing graph foundation models often require training additional classification heads or feature projectors to accommodate new graphs or new labels, whereas Acacia does not. Moreover, existing graph foundation models often gain their capabilities by being stitched together with pretrained LLMs, whereas Acacia is trained from scratch using only the Common Crawl web graph. This is also an important result because it provides evidence that graph models can acquire emergent capabilities from scratch like LLMs.