Potarcin tests abstract reasoning skills beyond simple output accuracy
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Artificial Intelligence
Summary
Current AI tests using the Abstraction and Reasoning Corpus (ARC) only check if a model can produce the right answer for specific inputs. This paper introduces PotARCin, a new way to evaluate AI that looks at different skills like understanding rules, classifying tasks, creating new examples, editing, and reversing problems. The authors found that many strong AI models do much worse when tested this way, showing they may not truly grasp the abstract rules. PotARCin also shows that models often contradict themselves, even when they can express the rule correctly.
What this means in practice
- •For ai developers: Evaluate AI models’ deep understanding of abstract tasks beyond simple answer correctness to better measure true reasoning abilities.
- •For computer vision teams: Use PotARCin’s expanded evaluation to test how vision models handle abstract rule changes and variations in input transformations.
Authors
Claas Beger, Ryan Yi, Melanie Mitchell
Abstract
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.