EnigmaForge challenges AI with hidden puzzles inside stories
EnigmaForge: The Question Is Hidden in the Story
Artificial Intelligence
Summary
Many AI tests give a question and ask for an answer, but EnigmaForge hides a puzzle inside a stack of old documents with no explicit question. The authors designed puzzles that only have one correct solution, confirmed by a type of logic solver. Different AI models showed very different abilities to solve these puzzles when just given the story, with some models even doing better without seeing a question. The work shows how puzzles hidden in stories create a new way to test how well AIs understand and reason about information.
What this means in practice
- •For ai model developers: Evaluate and improve AI reasoning skills by testing models on puzzles hidden in document stories rather than explicit questions.
- •For natural language processing teams: Use continually generated, uniquely solvable puzzles embedded in text documents to benchmark and refine understanding beyond surface fact retrieval.
Authors
Daniel Eisner
Abstract
Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and logbook margins is a small logic puzzle whose solution is unique - proved by a SAT solver at generation time, with an ablation certificate showing every clue is load-bearing. Because instances are generated rather than collected, the corpus renews forever. The headline measure is intuition: task success when handed only the story, with world reconstruction as the secondary axis. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions. Intuition reshuffles the leaderboard: a 22x spread where fact recovery spans 1.6x, the second-best fact-recoverer ranks fourteenth, one model is indifferent to being told the question, and another is significantly better without it. Several models were blocked by their own content filters before reaching the puzzle - any benchmark scoring refusals as failure is quietly measuring filter behavior.