Ai systems tested for discovering new rules in alien worlds
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
Artificial IntelligenceComputation and Language
Summary
It is hard to measure if AI can discover genuinely new things or just remember what it already knows. The authors created ExplorationBench, a setup with made-up alien environments where AI must explore and learn unknown rules. These environments let testers check answers exactly and make sure AI isn’t just recalling prior knowledge. They tested 10 AI systems and found some can learn new rules, but progress can vary and sometimes even backslide. This work helps evaluate how well AI can explore and learn in truly unfamiliar situations.
What this means in practice
- •For ai developers: Assess AI systems’ ability to learn unknown rules in controlled alien-world scenarios for debugging and improvement.
- •For game designers: Use verifiable alien-themed rule sets to create puzzles that test AI agents’ capability to explore and adapt.
Authors
Ming Zhang, Zhenghao Xiang, Peizhong Gao, Yujiong Shen, Yuhui Wang, Zhonghan Yue, Shihan Dou, Zhangyue Yin, Junjie Ye, Shichun Liu, Weihuang Zheng, Jiahao Chen, Jiayi Chen, Hongzhang Liu, Jiaqi Shao, Tao Gui, Qi Zhang, Xuanjing Huang, Suncong Zheng, Maxm Pan
Abstract
Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data. To this end, we introduce ExplorationBench, which turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds: their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. The benchmark contains two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks. We evaluate 10 AI systems and find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains. ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.