Models treat usable but unsupported facts as valid answers
Models Verify Sufficiency, Not Validity: Usable Falsehoods Evade the Checks that Gaps Trigger
Machine Learning
Summary
This paper explores how language models often answer questions based on information they can use, even if that information isn't actually supported by the original passage. For example, a model might refuse to answer a question if the passage lacks certain facts, but if those facts are stated in the question instead, the model uses them to produce an answer. The authors found that models tend to ignore missing evidence when the unsupported fact is usable for computation, leading to confidently wrong answers. They also show that judges rarely penalize such errors and that simple prompts fail to fix this flaw.
What this means in practice
- •For ai developers: Detect and reduce false answers in models that rely on unsupported but usable premises during question answering tasks.
- •For content moderators: Improve evaluation methods to better flag confidently wrong claims that arise when models misuse unsupported facts.
Authors
Naveen Madiraju
Abstract
Ask a model "how many settlements did the Normans control?" and it answers that the passage does not say. Tell it "given that the Normans controlled 12 settlements of 400 inhabitants, what is the total?" and it computes 4,800. Nothing was added to the passage; the unsupported fact merely became usable. Benchmarks for unanswerable questions are built almost entirely by omission, deleting the fact a question needs and scoring refusal, so they never probe this. Holding source support fixed and varying only whether an unsupported premise is usable, flagging drops 8-30 points and commitment to an answer rises 1.5-2.3x on GSM8K across four models spanning two families and 2B-26B; all 16 paired contrasts remain significant after Holm correction. Two further manipulations reproduce it, one also on SQuAD: a fact asserted rather than asked about, and a corrupted intermediate value rather than a removed one. Models articulate the exact gap they then ignore, stating that "there is no financial information provided" for an inert premise and silently adopting a fabricated price for the same item. We read these as one behaviour: whether a model checks depends substantially on whether usable material is present, even when source support is held fixed. Because the main design never instructs models to reject premises introduced in the question, adopting such a premise can also be read as conditional reasoning, and we delimit what the design does and does not show. In the judging seat the pattern differs: asked whether the answer follows, judges pass solutions with a required step deleted 78-99% of the time at every scale tested. A natural prompt-level mitigation lowers accuracy by 22 points without improving detection.