How agent harnesses affect success when context is lost
When the harness decides success: state that cannot be re-observed
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
This paper looks at how the way a language model is given information over multiple steps can change whether it succeeds or fails. The authors found that if important facts are removed and cannot be seen again, even simple reasoning models fail completely. They also show that summarizing past information instead of dropping it helps recover much of a performance gap between different methods. Finally, the advantage of better methods grows in longer, deeper tasks, though this is mostly because they keep going further.
What this means in practice
- •For ai system developers: Design agent systems that maintain key facts by summarizing instead of dropping old context to improve reasoning success.
- •For game ai developers: Build better game-playing AI by managing how past moves and reasoning are presented to models, improving performance on complex tasks.
Authors
Raimondas Lencevicius
Abstract
Agent harnesses decide what a language model sees across turns: they drop old turns, summarize them, or replay the model's own earlier reasoning. We ask when these choices change whether an agent succeeds, not only what it costs, and report three findings. First, in a pre-registered synthetic probe, truncating the context window costs a low-cost reasoning model nothing in success when a needed fact can be looked up again (it looks the fact up instead), but drops its success to chance when the fact cannot be re-observed: the two penalties differ by 92.5 percentage points (90% CI [89.5, 95.5]). A pre-registered replication on one model from a second provider showed the same contrast. Once the fact was lost, that model chose no answer rather than guessing. Second, on ARC-AGI-3, where the benchmark's Standard harness trails its Provider Adapter harness by a wide margin, we decompose that gap one component at a time, for one model on four games. Summarizing old turns instead of dropping them, while keeping the Standard harness's invitation to take notes, recovers 86% of a 63-point gap without replaying the model's reasoning. The same harness without the invitation recovers 43%, and adding the provider's native replay of the model's reasoning recovers the remaining 57%. These are point estimates from one counted run per configuration and game, and the replay share depends on a single run. Third, in the benchmark's published runs and in ours, the Adapter's advantage in the probability of clearing a level it has reached is larger at deep levels than at early ones. This largely restates where runs stopped and does not separate a growing advantage from a constant one.