HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
2026-06-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created HERO'S JOURNEY, a set of tests where machines try to figure out hidden rules from examples and then follow those rules step-by-step. They tested advanced language models and found that while these models can learn some rules, they struggle, especially with tasks needing multi-step actions. Changing how the rules looked didn't affect performance much, but the step-by-step execution was a big challenge. Techniques to help models learn rules worked better for simple tasks but not for more complex procedural ones, showing there's still work to do on those harder problems.
rule inductiongoal-directed tasksepisodic taskslanguage modelsmulti-step executionattribute inductionprocedural inductionbenchmarklexical groundingmodel evaluation
Authors
Anshun Asher Zheng, Kanishka Misra, David I. Beaver, Junyi Jessy Li
Abstract
We introduce HERO'S JOURNEY, a benchmark for rule induction in goal-directed episodic tasks, where agents must infer hidden rules from demonstrations and act on them through multi-step execution. HERO'S JOURNEY covers eight tasks across attribute and procedural induction families, each with four structural rule forms, controllable lexical grounding, and identifiability conditions. Evaluating state-of-the-art LLMs, we find that models show evidence of rule induction, but the ability is limited and uneven across tasks. Meanwhile, process execution adds an execution bottleneck for models, whereas surface semantics has minimal effect. Induction-specific steering methods improve performance on attribute tasks but show no reliable gains on procedural tasks, suggesting the gap in procedural induction remains an open challenge.