Large language models improve reasoning with semantic abstraction framework
Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models
Computation and Language
Summary
Large language models (LLMs) sometimes struggle to understand deeper meanings between sentences during tasks like language inference. The authors propose a framework that reorganizes word relationships to help these models reason more like humans by discovering missing knowledge. Their approach boosts model accuracy by over 10%, especially in understanding when statements do not follow from each other. This method focuses on structured knowledge rather than just adding more data, helping LLMs fill in gaps in their reasoning.
What this means in practice
- •For nlp engineers: Enhance natural language inference systems by integrating semantic abstraction to improve understanding of indirect or implicit relationships.
- •For chatbot developers: Improve chatbot reasoning abilities regarding contradictory or unrelated statements using new semantic knowledge frameworks.
Authors
David Torres-Moreno, Jorge Hermosillo-Valadez
Abstract
Despite their outstanding performance on many NLP tasks, LLMs face serious challenges related to semantic abstraction. In this study, we are interested in understanding how LLMs leverage abstract semantic knowledge in natural language inference (NLI), which requires sophisticated linguistic capabilities to interpret implicit meanings, contextual conceptual relationships, and semantic connections between words and phrases. To this end, we propose a methodological framework for constructing new semantic knowledge at a higher level of abstraction, which we define under the notions of semantic compatibility and incompatibility for NLI. In this framework, the meaning of the lexical-semantic relations between the premise and the hypothesis is reconfigured to achieve a more flexible semantic network that induces different reasoning paths in LLMs. These new pathways show a consistent pattern of responses that allows agreement on a single response. The results demonstrate that our proposal allows to discover and compensate for LLMs' semantic knowledge gaps in NLI, achieving significant improvements in accuracy, exceeding 10% for some models, and in particular for the non-entailment class. It is essential to note that LLMs need structured knowledge and not just more data to bridge reasoning gaps. Our hybrid approach directs attention to overlooked word relationships, allowing models to synthesize missing information. We believe that the future lies not in increasing model size, but in creating a semantic scafolding that mimics the flexibility of human thinking. Hopefully, our proposal will enable the development of more robust agents and interpretable reasoning, guiding AI toward reliable language understanding.