Mismatch in testing repairs can reduce AI agent performance

Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents

Software EngineeringComputation and Language

Summary

When AI agents powered by large language models (LLMs) fail, developers try to fix them by changing parts of their behavior. The paper shows that the way these fixes are tested can matter a lot—using a different or stronger AI to check repairs often chooses fixes that don’t help the AI actually running the system. The authors measured how much value is lost when repairs are picked by a model other than the one that will execute them. They recommend always testing repairs with the actual AI agent that will use them, and marking repairs to specify which agent they were tested on.

What this means in practice

  • For ai system developers: Ensure that fixes to AI agent failures are evaluated with the same model that will deploy them to avoid performance loss due to evaluator-executor mismatch.
  • For chatbot platform engineers: Improve chatbot repair selection by verifying candidate repairs with the deployed model instead of stronger or different models for better end-user outcomes.

Authors

Yijun Liao, Fanwei Liang

Abstract

When an LLM agent fails, repair methods choose which action to replace and what to put in its place. They often score candidate repairs with greedy replay, a stronger model, an environment oracle, or a judge that reads the failed log. We ask whether these evaluators pick repairs that help the agent that will actually execute them. Holding failed trajectories and candidate single-step repairs fixed, we label every candidate with independent rollouts of the deployed policy and measure how much deployment value each evaluator's choice loses. On failures not screened for repair value, pre-specified tests show that greedy replay loses 0.08-0.13 in expected net success gain in ALFWorld and 0.06-0.13 in net reward in WebShop, relative to selection with the deployed agent's own rollouts. At an equal rollout budget, selecting repairs for Qwen2.5-7B with Qwen2.5-14B rollouts loses 0.25 in ALFWorld and 0.07 in WebShop, twenty 14B rollouts per candidate are worth no more than two of the 7B agent's own, and fresh rollouts confirm that the two models prefer opposite repairs on selected disagreement cases. Between similar-sized models from different families, the losses appear in both directions in both environments (0.07-0.17). The scoring executor also changes repair-location labels and method rankings. Repairs should be verified with the executor that will run them, and repair labels should name their executor.