Benchmark evaluates knowledge graph building from semi structured data
Benchmarking Automated Knowledge Graph Construction from Semi-Structured Data
Databases
Summary
Building knowledge graphs, which are structured ways to store and represent information, is usually very time-consuming. The authors of this paper focused on automating this process when the input data is semi-structured, like tables or forms, rather than plain text. They developed a benchmark that tests how well different systems build knowledge graphs by checking things like correctness, completeness, and how well the graph answers specific questions. To support this, they created datasets across various topics and a set of measures to fairly compare different methods. This helps improve and reliably evaluate tools that create knowledge graphs automatically.
What this means in practice
- •For data engineering teams: Evaluate and compare tools that automatically create knowledge graphs from semi structured inputs using a shared benchmark.
- •For enterprise knowledge management: Improve automated population of enterprise knowledge graphs to support smarter data retrieval and decision making.
Authors
Tarek Al Mustafa, Birgitta König-Ries
Abstract
Knowledge Graphs (KGs) play an increasingly important role in numerous applications ranging from traditional knowledge representation to serving as memory for LLMs to support downstream tasks. However, their construction is labor-intensive; thus, in recent years, numerous approaches for automizing this process have been proposed. Compared to the popularity of construction approaches that focus on textual input data, methods for semi-structured inputs remain underrepresented and as a result, no comprehensive benchmark and evaluation suite exists to judge the quality of mapping predictions and generated KGs. This is problematic, as a KG's quality has direct influence on the downstream applications it supports and thus, strong evaluation mechanisms for their construction are urgently needed. In this work, we thus focus on the evaluation of KG construction from semi-structured data and present a benchmark and evaluation pipeline for KG construction that combines the quality dimensions (1) syntactic validity, (2) semantic accuracy, (3) consistency, (4) conciseness, (5) completeness, and (6) pragmatic quality measured on a KG's ability to provide answers to competency questions. This work contributes a realistic task definition, extends current state of the art evaluation frameworks, allows evaluation of systems that predict mappings and RDF data alike, and combines evaluation of both KG construction and downstream usage. We provide a comprehensive metrics suite, provide ten expert-curated datasets from seven domains, and showcase evaluation using two reference systems.