Large language models get better tested for real understanding of context
Evaluation of Contextual Understanding in Large Language Models
Computation and LanguageMachine Learning
Summary
Measuring how well large language models understand context is hard with usual tests that look only at simple accuracy. The authors developed a new way to check if these models really grasp and connect information by comparing how similar their answers are to real knowledge graphs, which map facts and their relationships. They created a score called Semantic Structural Similarity for Knowledge Graphs (S3KG) that looks at both meaning and structure in answers. Their tests show that this new method can better identify errors and how faithful the model’s responses are to the context, making evaluations clearer and more reliable.
Large Language ModelsContextual understandingKnowledge graphsSemantic similarityStructural similarityQuestion answeringEvaluation metricsReasoning errorsPerplexityBLEU score
Authors
Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne
Abstract
Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.