Reinforcement learning gains split into reaching and solving tasks
Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs
Artificial Intelligence
Summary
When AI agents learn to complete tasks step-by-step, it's unclear whether improvements come from reaching the right situations or handling them better. This paper introduces a method called checkpoint handoff, which lets researchers test these parts separately by swapping the starting points between different trained agents. The authors show that the gains from reinforcement learning come from both arriving at helpful states more often and solving from those states more successfully. This approach helps clarify what aspect of training leads to better overall performance.
What this means in practice
- •For ai development teams: Diagnose whether improvements in task completion come from better reaching or better solving to guide RL training strategies.
- •For game ai designers: Evaluate AI agents by separately testing how well they navigate to key game states and how well they finish tasks from those points.
Authors
Xuan Liu, Jingbin Qian
Abstract
Reinforcement learning now trains language-model agents that act over dozens of steps in live environments. The gains are large, and they are read as better decision-making. An agent in a closed loop writes its own inputs. Each observation follows from its own earlier actions, so the states it meets late in an episode are partly of its own making. An SFT checkpoint and an RL checkpoint are then scored from different states, even on identical tasks. Endpoint success mixes two changes: where the agent arrives, and what it does once it is there. Restricting the comparison to states both policies reach does not separate them. That restriction selects on an outcome, and in our data it flips the sign of the effect. We introduce checkpoint handoff, an evaluation protocol that clones a state one released checkpoint reached and hands it to another, with no retraining. Crossing a reacher role and a solver role over SFT and RL splits an endpoint gain into REACH and SOLVE. REACH is how often a policy arrives at a state the environment confirms is a fixed number of actions from success. SOLVE is how often it finishes from an identical cloned state. Across two benchmarks and two independently released pipelines, the reacher by solver interaction is positive in all five conditions. An RL history is worth more to an RL solver than the same history is to an SFT solver. On ALFWorld, RL improves both terms, and the SFT solver never succeeds where the RL solver fails. Independent REACH and SOLVE gaps predict the aggregate interaction. Handoff asks only that one checkpoint's history can be replayed under another, so long-horizon evaluation can report arrival and completion beside endpoint success.