Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?
Artificial Intelligence
Summary
The authors studied four versions of an AI agent designed to solve logic puzzles called ARC-AGI-3 games, each with different features like text-based reasoning or exact replay verification. They tested these agents using increasingly advanced AI models and more reasoning effort. They found that stronger models and more thinking time generally lead to better results, but the benefits of specific features depend on the setting. The version requiring exact replay verification performed best but used more resources. Their top agent solved all public games nearly perfectly, though this reflects only known puzzles and not new challenges.
Authors
Sergey Rodionov
Abstract
Our previous ARC-AGI-3 agent bundled executable world modeling, scheduled simplification, and exact replay verification, leaving unclear which idea accounted for its performance. We address this attribution question with four nested Codex-based agents: a textual baseline; a flexible-interface executable world model without replay verification; the same executable model with scheduled simplification; and a fixed-interface verification treatment that retains simplification and requires exact reproduction of recorded observations. The main study evaluates all four agents with gpt-5.4 and gpt-5.5 at high and xhigh reasoning effort on the public ARC-AGI-3 games. Exploratory follow-ups evaluate the textual and verification variants with gpt-5.6-sol at xhigh and max. The most robust result is that every agent variant improves with a stronger model and with greater reasoning effort. Within each model-effort setting, differences among variants are smaller than anticipated, while the effects of individual components vary across settings. Requiring a persistent executable deliverable is not universally beneficial: the textual variant outperforms the flexible-interface executable variant in both gpt-5.5 settings. Simplification improves performance in three of the four model-effort settings, with the weakest setting as the only exception. The complete verification treatment ranks first in all four settings, although it uses substantially more resources. In the gpt-5.6-sol follow-up, the verification variant fully solves every public game at both reasoning efforts, achieves about 99% RHAE, and uses fewer than half the total actions of the human baseline. Because the model postdates these games and held-out performance remains untested, this result should be interpreted as saturation of the public set only.