Ai reasoning needs tests for adapting to new problems and evidence
Beyond Answers, Traces, and Efficiency: Evidence for Adaptive Reasoning
Artificial Intelligence
Summary
Current AI systems are judged mostly by whether they get the right answer or show their steps, but this doesn't tell us if they can adjust their thinking when things change. The authors point out two types of adaptability: using what you learned in one problem to help in a new one, and changing your guesses when new information shows up. They suggest new tests that directly check if AI systems can transfer ideas across tasks and revise their hypotheses as they get new feedback. This approach can better measure how reliable and flexible AI systems are when facing unexpected situations.
What this means in practice
- •For ai system developers: Design evaluation tests that measure how well AI systems adapt their reasoning to new tasks and update hypotheses when given corrective feedback.
- •For software testers: Create testing frameworks that check AI responsiveness and robustness under changing inputs and evidence during runtime.
A position paper. It proposes an approach and reports no results.
Authors
Sejin Kim, Sundong Kim
Abstract
Current AI systems are evaluated as reasoners through final-answer accuracy, reasoning traces, process supervision, and computational or human-relative efficiency. Yet these measures do not establish whether reasoning adapts when tasks or evidence change. We distinguish across-task adaptivity, where acquired abstractions transfer or recombine across novel problems, fromwithin-task adaptivity, where hypotheses and actions are revised in response to new observations, feedback, or counterexamples. Using ARC-AGI as a diagnostic case study rather than an alignment benchmark, we examine what existing measures leave unresolved when test-time search, sampling, or adaptation affects performance. We propose intervention-oriented evaluation that tests abstraction transfer and hypothesis revision under controlled changes to tasks and evidence, with explicit reporting of test-time procedures and budgets. This perspective supports evaluating systems for robustness, responsiveness to corrective feedback, and auditability under changing conditions.