Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning
2026-05-11 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce GTLM, a new model that lets large language models (LLMs) understand graphs directly without squishing complex graph details into single tokens. GTLM adds a small number of parameters by adjusting the model’s attention to be aware of graph structure, which keeps the original model’s properties intact. They show that a much smaller GTLM performs as well as much larger existing models on standard graph tasks and even better on question-answering with graphs. The model’s attention mechanism also mimics how graphs pass messages, helping it handle algorithm-like reasoning within LLMs.
Large Language Models (LLMs)Graph Neural Networks (GNNs)Graph Transformer Language Model (GTLM)Attention MechanismNode Permutation EquivarianceText-Attributed GraphsMessage PassingAlgorithmic ReasoningGraphQAParameter Efficiency
Authors
Dario Vajda
Abstract
Using Large Language Models (LLMs) to process graph-structured data is an active research area, yet current state-of-the-art approaches typically rely on multi-step pipelines with Graph Neural Network (GNN) encoders that compress rich textual attributes into solitary tokens, creating a significant semantic bottleneck. In this paper, we introduce the Graph Transformer Language Model (GTLM), a novel architecture that enables pretrained LLMs to natively process graph topologies while entirely eliminating this compressive bottleneck. GTLM is exceptionally parameter-efficient: by injecting graph-aware attention biases directly into the LLM's attention modules, it introduces only 0.015% additional parameters relative to the base model. We theoretically prove that our bidirectional attention prefix preserves node permutation equivariance while maintaining exact backward compatibility with the pretrained base model. Extensive evaluations demonstrate that a 1B-parameter GTLM matches or exceeds the performance of 7B-parameter state-of-the-art models on standard Text-Attributed Graph benchmarks, while significantly surpassing baselines on GraphQA. Finally, we demonstrate that GTLM attention heads implicitly learn to simulate message passing, explaining its superior performance on algorithmic tasks. This paradigm shift enables true algorithmic reasoning within LLMs and provides a scalable foundation for next-generation GraphRAG and relational deep learning.