EviScope reveals when language models use wrong or missing evidence
EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models
Computation and Language
Summary
Answering questions correctly is not enough if a language model does not use the right evidence or ignores conflicts in information. The authors created EviScope, a test that changes the evidence around a fixed question to see if models stick to truthful support or get confused. They found models often answer correctly while ignoring contradictory or missing evidence, which challenges how we judge their trustworthiness. EviScope helps spot when models give unsupported or confused answers rather than just judging if the answer is right.
What this means in practice
- •For ai system developers: Detect when language models produce answers unsupported by or conflicting with their retrieved evidence to improve model trustworthiness.
- •For content moderation teams: Evaluate AI outputs for unsupported or contradictory claims to reduce misinformation in generated text.
Authors
Suryadeep Singh Deswal
Abstract
Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory. We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, distracting, or contradicting its evidence. EviScope-v1.1 contains 40 four-condition quartets with repaired counterfactual claims and span-level support labels for automatic evaluation. Across 960 gold-blind generations from Qwen2.5-7B, Llama 3.1 8B, and Gemini 3.5 Flash, paired metrics expose model-dependent grounding behavior that answer accuracy hides. On two local open models, an explicit evidence-action gate underperforms vanilla RAG on QCS: 0.15 vs. 0.50 for Qwen and 0.10 vs. 0.375 for Llama. Gemini reaches 0.944 joint success under both prompts, yet still answers 5% of conflict cases after contradiction insertion. EviScope therefore distinguishes unsupported answering, conflict blindness, and wrong non-answer actions rather than scoring answers alone.