Large language models struggle to follow different types of negation reasoning
Not What You Meant: Can LLMs Follow a Specified Negation Semantics?
Artificial Intelligence
Summary
Negation—the way 'not' is understood—can mean different things depending on context, like in law or medicine. The authors tested whether big language models naturally understand these different meanings or can adjust when told how to interpret negation. They created a set of logic problems with known answers and turned them into sentences for the models to solve. Results show that the models only partially get the correct interpretation and often change answers if the problem wording changes, showing inconsistency. Some newer models do better but still don’t fully solve the problem, though methods like using specialized solvers or extra training help somewhat.
What this means in practice
- •For legal tech developers: Improve AI tools to better handle legal negation by recognizing different logical readings of negation in rules and regulations.
- •For medical data teams: Build systems that correctly interpret negation in medical guidelines, reducing errors in automated medical reasoning.
Authors
Qiming Bao, Agnieszka Mensfelt, Michael J. Witbrock, Kostas Stathis
Abstract
Negation does not carry a uniform interpretation across domains. In legal, regulatory, and medical reasoning, the intended interpretation depends on the reading in force -- open- versus closed-world, two- versus three-valued, and credulous versus skeptical. We study which reading of negation large language models adopt by default and whether they can override that preference when a different reading is explicitly specified. To this end, we introduce NAFBench, a procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics (WFS), and credulous and skeptical reasoning under stable-model semantics. The generator emits ground normal logic programs with controlled depth, width, and cycle structure. Each program is solved under all four viewpoints using SWI-Prolog, a well-founded semantics solver, and clingo, yielding up to four divergent labels. The programs are then verbalized into natural language under multiple framings and rule orderings that leave the answer invariant. The results expose a consistent gap. Across open-source models, following a specified negation semantics remains unsolved: the strongest models score 59--74% across the four semantic viewpoints, while the weakest score 31--67%. All models are order-sensitive on more than half of logically identical rule shufflings, while the two weaker models frequently overcommit on well-founded "undefined." Two frontier models reach 100% on the main fixed-complexity evaluation set, and a third, o4-mini, is near-perfect, falling only to 81% on well-founded "undefined." Delegating reasoning to a solver, fine-tuning on certified traces, or forcing an explicit three-valued verdict each partly closes the gap.