AI summaryⓘ
The authors investigated how language models reason about necessity and possibility, which rely on assumptions about connections between possible worlds and objects in each world. They created pairs of problems that look the same but differ in underlying conditions, and used automated tools to confirm that the correct answers differ. They found that most models struggled to follow the precise semantics unless put into a special reasoning mode, which dramatically improved one model's performance. They also observed that when details about the world connections were missing, models agreed but seemed to use simpler logic rules instead. The authors provide all problem materials and results for others to explore.
modal logicpossible worldsaccessibility relationframe conditionsdomain conditionslanguage modelsautomated reasoningnecessitypossibility
Authors
Réemi Andrieu, Damien Sileo
Abstract
Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.