TaLK: Text-attributed Graph Dataset Distillation via Coupling Language Model with Graph-Aware Kernel
2026-06-22 • Machine Learning
Machine Learning
AI summaryⓘ
The authors focus on text-attributed graphs (TAGs), which combine text and graph information. They point out that training models to handle both aspects together is very slow and costly. To fix this, they introduce TaLK, a new method that speeds up learning by using a clever kernel approach instead of repeatedly training complex models. Their experiments show TaLK can perform almost as well as using all the data but with only a tiny fraction of it. This makes learning on TAGs much faster and more efficient.
text-attributed graphslanguage modelgraph neural networkdataset distillationneural tangent kerneljoint trainingtext semanticsgraph structure
Authors
Yeongho Kim, Yeonje Choi, Kijung Shin
Abstract
Text-attributed graphs (TAGs) are widely used in many real-world domains, and learning on TAGs requires jointly modeling text semantics and graph structure. A standard approach for modeling TAGs is to combine a language model (LM) and a graph neural network (GNN), but joint training is computationally expensive and difficult to scale. Dataset distillation is a promising way to reduce training costs, but existing methods are not well suited to TAGs because they are typically designed for a single modality or still require repeatedly training expensive LM-GNN models on the full dataset during distillation. To address this, we propose TaLK, an effective dataset distillation method for TAGs that couples an LM with a graph-aware neural tangent kernel.This design enables efficient dataset distillation, avoiding repeated joint training on the full dataset while reflecting both textual and structural information for effective TAG learning.Experiments on multiple TAG benchmarks show that TaLK consistently outperforms existing baselines and achieves up to 97% of full-dataset performance with only 1% synthetic data.