LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs
2026-04-09 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how well large language models (LLMs) identify relationships between things in sentences compared to a smaller, graph-based parser. They found that when sentences have more complex relationships, the graph-based parser does a better job than the large models. This suggests that simpler, specialized tools might work better than big models for tricky language structures. They tested this across six different datasets to come to their conclusion.
relation extractionknowledge graphslarge language modelsgraph-based parserlinguistic graphsupervised learningin-context learningsentence graph complexity
Authors
Paolo Gajo, Domenic Rosati, Hassan Sajjad, Alberto Barrón-Cedeño
Abstract
Relation extraction represents a fundamental component in the process of creating knowledge graphs, among other applications. Large language models (LLMs) have been adopted as a promising tool for relation extraction, both in supervised and in-context learning settings. However, in this work we show that their performance still lags behind much smaller architectures when the linguistic graph underlying a text has great complexity. To demonstrate this, we evaluate four LLMs against a graph-based parser on six relation extraction datasets with sentence graphs of varying sizes and complexities. Our results show that the graph-based parser increasingly outperforms the LLMs, as the number of relations in the input documents increases. This makes the much lighter graph-based parser a superior choice in the presence of complex linguistic graphs.