Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge
Computation and Language
Summary
The authors created ElephantBench, a test set of questions designed to show when large language models (LLMs) remember different, conflicting facts instead of just one answer. They made this by finding real disagreements in sources and turning them into questions with multiple answers, all checked carefully by people. When testing 32 models, even the best could only remember both answers about half the time, often missing one. Larger models and more reasoning helped a bit but didn’t solve the problem. Their work highlights how models might overlook less common facts and offers a way to better check this using a special graph-based method.
Factual question answeringLarge language modelsLong-tail factsKnowledge probeGraph-based pipelineParametric memoryEpistemic myopiaMulti-account QAModel scalingInference-time reasoning
Authors
Zhuoshi Pan, Junru Lu, Yan Qian, H. Vicky Zhao, Di Yin, Xing Sun
Abstract
Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.