Papers for

interactive chatbot teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Kepler tests AI game-playing models by checking their understanding

Kepler: Auditable World Models for ARC-AGI-3

Abstract: ARC-AGI-3 evaluates agents in interactive environments whose rules and objectives must be inferred from observation. We present Kepler, an open-source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. Under one frozen Claude Opus 5 configuration, Kepler obtained a server-verified 100.00 RHAE on all 25 public games, with no per-game model selection or score-conditioned reruns. On 181 of 183 completed levels, the final Opus attempt used no more actions than the corresponding median-human baseline. The retained board runs used 8,256 environment actions, of which 7,292 occurred in scored levels. Retained local provider-session records yield 858.0 million tokens, 97.37% cache reads, and a \$777.72 cost at September 1, 2026 API list-equivalent rates. We also report three evaluation failures: source-code leakage that produced an invalid perfect run, agents reconstructing a removed harness in a control condition, and autonomous repair masking a broken planner. A single-game observation case study showed that animation frames contained task-relevant information absent from settled text grids. Across the final Claude Opus 5 and GPT-5.6 Sol boards, 48 of 50 game-model cells reached 100. These results indicate that public-set score alone has limited discriminative value and motivate first-attempt, cost-conditioned, and verification-aware reporting.

Wed 30 SeptArtificial IntelligenceMultiagent Systems
The gist
Games with unknown rules are hard for AI to master because the AI must figure out the rules and goals just by playing. The authors introduced Kepler, a system that builds and runs possible game rules as computer programs to see which ones explain the game best. Kepler tested these hypotheses by checking if the models predicted future game states correctly and matched past moves. This approach showed that simply scoring points in games doesn't fully measure how well AI understands the rules or tasks.
Open → 2610.00834v1