Efficient fusion of large language models and graph neural networks

DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion

Machine Learning

Summary

Combining large language models (LLMs) with graph neural networks (GNNs) can improve how computers understand and classify data linked in graphs, like social networks or molecules. But these two types of models learn very differently, making it hard to blend their knowledge effectively without slowing things down. The authors created DeKoR, a method that breaks down the knowledge from LLMs and transfers it efficiently into smaller parts, then selectively uses this information for each data point. Their approach works better and faster than previous methods in tests.

What this means in practice

  • For data engineers: Improve graph-based data classification in production systems by efficiently integrating language and graph model knowledge.
  • For ai platform developers: Build AI tools that combine textual and graph information for enhanced pattern recognition in complex network data.

Authors

Peiyuan Li, Yongqi Huang, Jitao Zhao, Di Jin, Dongxiao He

Abstract

Large Language Models (LLMs) have shown strong potential for text-attributed graph learning by leveraging rich pre-trained knowledge and language understanding capabilities. Combining LLMs with Graph Neural Networks (GNNs) offers a promising paradigm for expanding the modeling capabilities of graph learning. However, LLMs and GNNs follow distinct learning paradigms, leading to a pronounced mismatch between their knowledge spaces. Such discrepancies make effective knowledge fusion nontrivial. Moreover, existing methods often incur substantial additional computation for only limited performance gains. To address these challenges, we propose DeKoR, an efficient framework for GNN and LLM knowledge fusion. To enable efficient knowledge transfer, DeKoR decomposes LLM knowledge into complementary local and global views and uses the LLM as an offline teacher to distill them into lightweight experts. To enable selective knowledge fusion, an adaptive router learns how much each node should rely on each expert. Extensive experiments on multiple benchmarks demonstrate that DeKoR outperforms existing baselines on both node and subgraph classification, achieving superior effectiveness and efficiency.