Benchmark assesses large language models generating ontologies from questions
CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions
Artificial Intelligence
Summary
Creating structured knowledge bases called ontologies from questions people ask is important but takes a lot of work. The authors made a new test set called CQ4OE to fairly check how well large language models can turn these questions into correct ontology parts. They link each question to exact pieces in a high-quality reference ontology and offer two ways to measure success: one looking at specific terms and another at the whole ontology structure. Their tests with nine different language models show these models do better picking correct terms than building full, complex ontologies with relationships and rules.
What this means in practice
- •For ontology engineers: Use CQ4OE to systematically evaluate and improve LLM-based tools for ontology creation from competency questions.
- •For knowledge graph developers: Apply the benchmark to assess automated ontology generation methods that support structuring knowledge graphs from user queries.
Authors
Jiayi Li, Ziyuan Wang, Daniel Garijo, María Poveda-Villalón
Abstract
Ontology generation from Competency Questions (CQs) is a central yet labor-intensive phase of Ontology Engineering. While large language models (LLMs) offer promising automation capabilities, current evaluations remain fragmented. Task formulations are heterogeneous, gold standards often lack fine-grained CQ provenance, metrics conflate lexical overlap with structural and logical adequacy, and reference ontologies are not always explicitly designed around the evaluation CQs. Here, we address these limitations with CQ4OE, a benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from CQs. For each ontology in the benchmark, we build a CQ-driven gold OWL ontology with explicit provenance linking each CQ to the classes, properties, and axioms required to answer it. From this resource, we define two complementary evaluation tasks. CQ2Term supports term-level evaluation of CQ-specific class and property prediction over 99 CQs, and CQ2Onto supports ontology-level evaluation over 118 CQs, including hierarchy, property modeling, and axiom-level structure. We demonstrate CQ4OE with experiments using nine LLMs under zero-shot, iterative, and multi-agent generation strategies, showing that LLMs recover explicit vocabulary terms more reliably than creating ontologies, particularly in property modeling, hierarchy construction, and axiom generation.