Benchmark tests how AI finds clues and solves mysteries in stories
WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot
Computation and LanguageArtificial Intelligence
Summary
Figuring out how AI can find important clues and understand stories on its own is tricky because stories often hide key information or present it in tricky ways. The authors created a test called the WORTH Benchmark where AI models read mystery stories without any hints, try to pick out important clues, identify people involved, and explain their reasoning. They ran experiments with AI models on two story episodes and found the AI often made similar mistakes, picking the wrong suspects and clues. This shows that the way information is presented can affect how AI finds evidence and makes decisions.
What this means in practice
- •For mystery game developers: Test AI story analysis to improve non-player character reasoning and clue presentation in interactive mystery games.
- •For legal tech teams: Develop AI tools that can autonomously select crucial evidence and participants from long narrative documents without prior annotations.
Tested on one dataset.
Authors
Bo Zou, Chao Xu
Abstract
Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.