Large language models assessed with new context understanding test
Evaluation of Contextual Understanding in Large Language Models
Computation and LanguageMachine Learning
Summary
Large language models can do many language tasks well, but it is unclear how well they really understand the meaning behind the words and context. The authors designed a new way to check how well these models understand and use facts and relationships in context, using something called a knowledge graph. They created a new score, S3KG, that measures how closely the model's answers match the facts and structure of the knowledge graph. Their tests show that this method helps reveal how accurate and faithful the models are to the given information.
What this means in practice
- •For ai product developers: Use the S3KG evaluation to better measure how well language models ground their answers in given context during product testing.
- •For machine learning engineers: Diagnose and categorize reasoning errors in language model outputs to improve their contextual reasoning capabilities.
Authors
Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne
Abstract
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.