ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
2026-07-06 • Artificial Intelligence
Artificial IntelligenceSymbolic Computation
AI summaryⓘ
The authors created ClassicLogic, a new set of puzzles to test how well AI can learn and combine different problem-solving steps. It includes familiar logic games like Sudoku and KenKen, with a special system that breaks down complex strategies into simpler parts. This helps measure if an AI really understands and can use these strategies to solve harder puzzles. Their work aims to improve AI's reasoning abilities beyond just language tasks.
Compositional generalizationLogic puzzlesSudokuKenKenHierarchical knowledge baseProblem-solving strategiesNeuro-symbolic AIArtificial intelligence benchmarking
Authors
Mahnoor Shahid, Hannes Rothe
Abstract
Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.