SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
2026-06-08 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors revisit their method called SEF-CLGC, which mixes formal logic with small language models (SLMs) to test reasoning skills. They focus on a challenge from SemEval-2026 that looks at how language models handle content versus logic in reasoning. Their results show that using only SLMs trained on both natural and symbolic languages, their top model gets about 28% on the content score while reducing bias from content in reasoning.
Small Language ModelsFormal LogicReasoning PerformanceSemEval-2026 Task 11Content BiasSymbolic LanguageNatural Language ProcessingSyllogistic Evaluation FrameworkLogic Grammar
Authors
Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin
Abstract
This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.