Scoped semantic invariance improves consistency in large language models
Scoped Semantic Invariance as a Foundation for LLM Reliability
Artificial Intelligence
Summary
Large language models (LLMs) sometimes give different answers to questions that mean the same thing, even if they know the facts. The authors study consistency by looking at how model answers stay the same when questions are changed without changing their meaning. They find that checking this kind of consistency everywhere is impossible, so they focus on smaller, well-defined areas using knowledge graphs to guide model behavior. Their work shows that inconsistencies tend to cluster in certain patterns in these graphs, and that careful use of these graphs can make model answers more correct and consistent.
What this means in practice
- •For natural language processing engineers: Improve the correctness and consistency of language model outputs by selectively grounding them in structured knowledge graph domains identified as semantic hotspots.
- •For knowledge base developers: Detect and reduce inconsistent model answers around specific knowledge graph substructures to improve reliability during automated reasoning tasks.
Authors
Rosario Uceda-Sosa, Maria Chang, Karthikeyan Natesan Ramamurthy, Moninder Singh
Abstract
LLMs often answer semantically equivalent queries inconsistently, even when the underlying facts are known. We formalize consistency as semantic invariance over equivalence classes of queries and show that invariance under arbitrary meaning-preserving transformations is undecidable in general. This motivates scoped semantic invariance, where model behavior is evaluated relative to bounded semantic domains. We operationalize this framework using knowledge graphs (KGs), which act as explicit semantic control structures defining local semantic constraints and grounding contexts. Across the MITRE D3FEND ontology and eight Wikidata- derived domains, we show that inconsistencies are structured rather than random, concentrating in localized hotspot subgraphs organized around recurring TREE, STAR, and CYCLE patterns that remain stable across models and domains. We further show that selective graph grounding improves correctness and cross-model consistency, while indiscriminate grounding introduces regressions. A layered grounding strategy achieves the strongest monotonicity profile, preserving correct knowledge while reducing inconsistency. Finally, we show that hotspot structures discovered from simple consistency probes transfer to more realistic compositional reasoning tasks.